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ABSTRACT 

This handbook provides suagesticzis about quality 
control procedures that will enable Teacher Corps project personnel 
to assess the effects and success of educational productSr practicesr 
and other outputs prior to dissemination. Two assessment processes 
are outlined. The first r review^ is the collegial ongoiiivj process for 
Teacher Corps project personnel to assess the effects and potential 
impact of innovations before dissemination. The second r validation r 
is the formal post-development process for Teacher Corps projects to 
Jissess'tbe evidence of effectiveness for approval by an official 
panel. Guidelines and checklists for reviewing innovative products 
and practices are provided. The criteria used for validating an 
educational prograr. are presented along with descriptions of the 
responsibilities of proiect personnel . in the validating process. A 
format for s«bmitti-na materials to a dissemination review panel is 
included as well as a description of panel criteria used for judging 
program effectiveness. (JD) 
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FOREWARD 



Educators who consider adopting elements of the school improvement and 
educational professional development programs of Teacher Corps projects may 
reasonably expect evidence that a specific product, practice, or process 
really works. Teacher Corps project personnel will have to provide adopters 
with plausible information about their innovations that demonstrates both 
the credibility of supporting evidence and the educational significance of 
each product or practice. 

Current Teacher Corps regulations require projects to develop specific 
objectives to achieve the adoption or adaptation of processes, practices, 
and products found useful in project schools to: 

® Persons involved in the project; 

® Schools of the local educational agency and components of the 
institution of higher education which are not involved in the 
project; 

® Other local educational agencies, institutions of higher 
education, and communities; and 

® Others interested in educational policy. 

Toward this end, the staff of the Teacher Corps Dissemination Project has 
worked to identify quality control criteria for Teacher Corps projects to 
use in assessing the program elements that they choose to share with others. 

This handbook represents a great deal of collaborative interaction by 
the contractor's staff and Teacher Corps project personnel, regional network 
staff, the Research Adaptation Clustcnr, the national program office, ana 
specialists in educational assessment. The handbook, and the materials 
provided for projects In the Teacher Corps Dissemination Tool Kit, contain 
effective procedures and practical advise for project personnel striving to 
stimulate the adoption and adaptation of their innovations. 

Teacher Corps projects are developing a number of exciting educational 
innovations for children in schools serving low income families. We want 
you to extend the benefits of these programs to schools throughout the 
country. This handbook should prove useful to you in assessing the 
educational significance and effectiveness of the products and practices 
that you share with others. 
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PREFACE 



At a time when there are literally thousands of Innovative educational 
products and practices to choose among, the task of sorting through all of 
the possible alternatives Is a formidable one Indeed. The developers of 
Innovations have to be ready to respond to the questions of potential 
adopters, Including the following: 




"Yeah, but, how 
do you know it 
really worked?*' 



"Yeah, but, how 
do you know it 
will work here?" 



"Yeah, but, how 
do I know that 
those kids are 
like ours?" 



Teacher Corps project personnel, as do all developers of innovative 
programs, have an obligation to be prepared to respond to these and to 
similar questions. It is necessary to collect data on the effects of an 
educational innovation, to document carefully its development, adaptation, 
and institutionalization in order to be able to respond in professionally 
adequate ways to potential adopters. 
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This handbook provides Teacher Corps project personnel with general 
guidelines for assessing educational products and practices. It may be 
used in the preparation of detailed procedures for assuring the quality of 
project-developed Innovations locally prior to dissemination to other 
educators. In short, the handbook should make It more likely that Teacher 
Corps project personnel will be better able to respond to those "yeah, but 
, . ." questions as they disseminate their proven products and practices. 
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The handbook has been prepared to provide Teacher Corps project and 
national office personnel with Information about alternatives for assessing 
educational products and practices. It provides criteria and procedures 
for reviewing and validating educational products and practices developed by 
Teacher Corps projects, A field-trial version of the handbook has been 
revised on the basis of feedback from project personnel. 

We acknowledge the Insights and contributions provided In the thoughtful 
reviews and critiques of background papers for the handbook by David L. 
Clark of Indiana University, Roy A. Edelfelt of the National Education 
Association, David D. Marsh of the University of Southern California, and 
our colleagues at the Far West Laboratory for Educational Research and 
Development: Bela Banathy, Paul Chrlstensen, Paul T. Hood, Diane Mclntyre, 
and Fred Rosenau. 

We are grateful also for the Information and advice provided by many 
members of the Teacher Corps In meetings of the Board of Directors of the 
Rocky Mountain Network, the Southeast Network, and the Southwest Network; 
In meetings with the Board of Directors and staff of the Research Adaptation 
Cluster; In personal discussions with staff personnel of Youth Advocacy 
Loop and the Executive Secretary of ^he Texas Network; and In telephone 
contacts with all regional network Executive Secretaries, 

We particularly appreciate the advice and materials provided by Keith 
Acheson, Director, Teacher Corps Project at the University of Oregon; Sara 
R. Ma^^ey, Executive Secretary of the New England Network; Lee Morris, 
Executive Secretary of the Southwest Teacher Corps Network; Robert A. 
Mortenson, Executive Secretary of the Plains Network; Thomas Nagel , Director, 
Teacher Corps Project at San Diego State University; William M. Quirk, 
Director, Teacher Corps Project at Emporia State University; Robert Spaulding, 
Director, Teacher Corps Project at San Jose State University; and John Reed 
Williams, Executive Secretary of the California Teacher Corps Network. 
Susan L. Mel nick, the project monitor at Teacher Corps Washington, helped 
us a great deal in shaping the handbook. 

Editorial assistance was provided by Marilyn Woodsea and Julie 
McCullough; Ann Wallgren supervised the typing and layout of the manuscript. 
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INTRODUCTION 



Background 

Since it began operations in 1966, the Teacher Corps of the U.S. 
Department of Education has been a major influence in teacher education in 
the United States. Over the years, many innovative programs have emerged 
as Teacher Corps projects sought to carry out program mandates to strengthen 
. educational opportunities available to children from low-income families, 
to stimulate development of teacher preparation programs for teachers of 
children from low-income families, and to encourage institutions of higher 
education and local educational agencies to improve programs of training 
and retraining for teachers and teacher aides. 

The 1976 amendments to the Higher Education Act of 1965 reflect the 
intent of Congress to increase (1) attention projects will give to improving 
the school/learning climate, and (2) emphasis on reforming the training 
and retraining of educational personnel, and to provide (3) greater focus 
on demonstration, documentation, institutionalization, and dissemination 
of the outcomes of Teacher Corps projects. 

The current Rules and Regulations of Teacher Corps assign to projects 
the responsibility for taking steps to ensure the adoption or adaptation by 
other educational agencies and institutions of the products, practices, and 
processes that projects may develop. Teacher Corps project personnel are 
expected to develop objectives for stimulating adoption and adaptation of 

ERiC ^0 
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thelp ppsdoets and ppsetlees by dcfronstfating and dliscffilnatlng thwR to: 

1. Persons Involved In the project i 

2. Schools of the local educational agency and components of the 
Institution of higher education not Involved In the project; 

3. Other local educational agencies, Institutions of higher 
education, and communities; and 

4. Others Interested In educational policy. 

It Is to these outcomes and objectives that the Teacher Corps Dissemination 
Project addressed Its attention. The project had two principal objectives; 

1. To assist Teacher Corps In establishing an effective system 
for sharing Information about products, practices, and other 
outputs within Teacher Corps, among projects and project 
personnel. 

2. To design and pilot-test a process for validating and 
disseminating Teacher Corps products, practices, and other 
outputs to educational Instltjclons and agencies throughout 
the country. 
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D1»»ciinnot1qn; Sroatfty Concglvcd 

In this handbook, dissemination Is considered as a set of more or 
less discrete activities within the larger context of planned change In 
educational Institutions. Within this context the Voles'* of developer, 
documentor , evaluator, demonstrator, and so forth may seem to overlap with 
the activities of the disseminator. But In reality all are elenwnts of an 
Institutional configuration that enables Individuals, who have personal 
commltnent to school Improvement, to extend and deepen the Impact of their 
efforts. The planned change model , seen In the context of a Teacher Corps 
project. Includes alH the outreach activities that eventually lead to the 
adoption or adaptation of project-developed Innovations— products, processes, 
and practices—by other educational agencies and Institutions. 
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Thus, dissemination cannot be regarded as a separate task to be under- 
taken in the third or fourth year of a Teacher Corps project; it is an integral 
part of the process of planned change* Dissemination should be regarded as 
a two-way process-'-where the spread and exchange of information about educa- 
tional innovations can facilitate choice among alternatives and contribute to 
effective impl ementation--that can assist Teacher Corps projects extend the 
impact of their programs beyond their local sites. When viewed as an integral 
part of school improvement and educational personnel development efforts, 
dissemination emerges very early and continues throughout the life of a project 

° During the needs assessment phase project personnel identify 
"targets" for improvement; this process also identifies the 
potential "audiences'' for outreach efforts. 

° Preparing project objectives involves the delineation of 
baseline conditions that will serve to measure change and 
for making claims about program effectiveness. 

° A project's evaluation design provides parameters for defining levels 
of educational and experimental "significance", measures that 
provide data for establishing the plausibility of project effects. 

° Implementing a school improvement or educational personnel 

development program (which may utilize resources or materials that 
have been identified through educational dissemination systems) 
provides documentation that can be used to assess program cost 
effectiveness, estimates of "transportability", and data of program 
effects on students and educational personnel. 

° Institutionalization of project-developed innovations is, in 
effect, an outcome of dissemination efforts at the local level. 

° Outreach is the natural extension of the planned-change process. 
Although in its early stages outreach may not involve a great 
deal cf face-tG-TuCe contact with potential clients, it requires 
the same kincis of personal interaction and perseverence to help 
- adopters adapt project-developed products or practices that are 
needed for local institutionalization. 

This handbook- wil 1 help Teacher Corps project personnel assess the effects 

of project-developed products, processes, and practices so that they will be 

able to provide adopters with plausible evidence of the effectiveness of the 

innovations. 
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Promising, Successful, or Exemplary? 



The purpose of dissemination is to share information about successful- 
educational programs. Problems arise, however, when we try to define what 
we mean by "successful . The evaluation of educational outcomes can range 
from an intuitive sense that "something happened" to very sophisticated 
measures of change that involve statistical analyses and rigorous research 
procedures. This handbook provides suggestions about quality control 
procedures that will enable Teacher Corps project personnel to assess the 
effects--the success— of educational products, practices, and other outputs 
prior to dissemination. Two assessment processes are outlined in the 
handbook: 

° Review is the collegial on-going process for Teacher 
Corps project personnel to assess the effects and 
potential impact of innovations before dissemination 
among other Teacher Corps projects and other educational 
audiences (pages 11 to 33). 

° Val idation is the formal post-development process for 
Teacher Corps projects to assess the evidence of 
effectiveness for review by the Department of Education's 
Joint Dissemination Review Panel (JDRP) or state 
Identification, Validation, Dissemination (IVD) panels 
(pages 35 to 49). 



Each process will require local or regional 
adaptation when it is implemented. The hand- 
book suggests procedures and criteria which 
have general applicability for assessing 
educational innovations. In order to meet 
the particular requirements of each project, 
however, it will be necessary for Teacher 
Corps project personnel to develop specific 
procedures (forms, documentation requirements, 
etc.) to accomplish each of the processes. 
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In general, the processes have the following distinctions: 



Review 

WHO? Project personnel 

(others as appropriate) 



Val I'dation 
Project personnel and 
evaluation consultant (s) 



WHAT ? Innovations being 

developed or already 
implemented locally 



Innovations found effective 
in project school or college 
sites 



WHERE ? Local project 

LEA and IHE sites 



At local project sites; then 
forwarded for review by 
Teacher Corps Program office 
or state IVD panel 



WHEN? Continuously 



Whenever required 



HOW? As part of an ongoing 

evaluation; in preparation 
for disseminating to other 
Teacher Corps projects or 
other educational audiences 



As the formal process of 
prescreening innovations 
before submitting to the JDRP 
or state IVP panels 



WHY? To collect evidence of the effectiveness of project-developed 
products and practices to share with potential adopters. 



NIE Product Review Criteria 

The criteria suggested for use 1n the project review process have been 
adapted from those developed by the Dissemination and Resources Group of 
the National Institute of Education (NIE). These criteria, synthesized 
from the research literature on the evaluation, assessment, review, and 
selection processes for educational products and practices (ERIE Institute, 
1978), are described in Appendix A, pages 51 to 58, and are incorporated 
in the checklist suggested for use in reviewing innovations developed by 
Teacher Corps projects (pages 16 to 33)* 
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The JDRP and State IVD Processes 

In the U.S. Department of Education the JDRP is charged with applying 
quality standards for products and practices--criteria relating to educa- 
tional significance and credibility of supporting evidence--in the review 
of claims made for educational innovations. Excerpts from the JDRP Ideabook 
(Tallmadge, 1977) about JDRP criteria are included in the handbook section 
on validation. States variously make use of IVD panels or other review 
and certification procedures to identify effective products and practices. 

The JDRP meets periodically in Washington to review submissions from 

Department of Education program offices (such as Teacher Corps Washington). 

. . .which are responsible for identifying potentially exemplary 
projects and for carrying out a pre-review screening for factual 
accuracy, social fairness, and possible harm to users, as well 
as for evidence of effectiveness. Only those submissions that 
are approved by the JDRP may be in any way endorsed by the 
Education Division or disseminated as exemplary using Federal 
Education Division funds. (Tallmadge, 1977) 

The process outlined for project validation in this handbook will aid Teacher 

Corps projects make JDRP or state IVD submissions. 

It is not anticipated that all Teacher Corps products and practices 

will undergo JDRP or IVD review. Many of the products and practices developed 

by Teacher Corps projects will be better suited to other dissemination systems. 

Products that include copyrighted material, for example, cannot be reproduced 

for further distribution without the permission of the copyright holder. We 

have outlined the process of project review in this handbook to enable Teacher 

Corps projects and networks to get promising products and practices, and/or 

information about them, into the many channels for dissemination that exist 

in addition to the JDRP/NDN system (see Guidelines for Dissemination of 

Teacher Corps Products and Practices ). 



EKLC 



17 



- 9 - 



Exemplary Products 

The point to remember here is contained in the phrase "disseminated as 
exemplary using Federal . . . funds." A project £an use its funds to 
disseminate products and practices to other LEAs and IHEs--and provide 
information about the effectiveness of the innovations at the local 
site— as long as project personnel do not state or imply that the 
products or practices are "exemplary" or are "endorsed" by the Depart- 
ment of Education. That indication of official approval is reserved 
to the JDRP. 



Some of the important terms and concepts used in the handbook are 
defined on the following page. The next two sections of the handbook 
outline the procedures and criteria for reviewing and validating Teacher 
Corps products and practices. The processes are distinct in terms of the 
persons who serve in particular roles and in the emphasis on assessing 
practical outcomes and empirical evaluation data. 
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Educators who use this handbook need to be aware of the definitions 
of several terms from the field of dissemination as. they are used herein. 
The JDRP Ideabook provides the following definitions: 



Educational 
Innovation 



Educational 
Products 



Educational 
Practices 



A broad concept which includes products, 
processes, practices, programs, and other 
outcomes. 

"Products" and "materials" refer to trans- 
portable, tangible or packaged programs or 
practices such as curriculum materials, 
educational management systems, computer 
programs, staff training material, guides, 
models, administrative procedures and 
Project Information Packages (PIPs). 

"Practices" and "methods" refer to systematic 
programs, procedures, or techniques whose 
adoption or adaptation requires assistance 
from the originators or their representatives. 



This handbook also makes use of the more precise perspective on 
educational materials defined by Emrick and Peterson (1978): 



Materials : brochures, manuals, workbooks, handbooks, filmstripS: 
videotapes, and other hard-copy or mediated presentations of 
information. Three different types of materials are used in 
di s semi nation/util izat ion efforts : 

Descriptive materials : printed matter, visual displays, 
and other hard-copy information designed to communicate 
what the new knowledge, product, or practice is, how it 
can be used, and what benefits will accrue from use. 



Instructional materials : the textbooks, workbooks, 
audiovisual sequences, and other items which make up the 
basic curriculum or content of the educational process 
(curriculum materials that are not central to the 
innovation are classified as support materials). 

Support materials : printed matter, audiovisual aids, and 
other informational components that occupy a background or 
optional status; support materials include elements of an 
innovation's curriculum, management, and implementation 
that are neither central to the innovation nor essential 
to its utilization. 
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GUIDELINES FOR PROJECT REVIEW 
OF INNOVATIVE PRODUCTS AND PRACTICES 
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Local Quality Control 

The guidelines that follow provide suggestions for procedures and 

criteria for reviewing Teacher Corps products and practices. The process 

that is outlined here is very general; each project will have to determine 

the specific documentation requirements and the nature of the evaluation 

processes for assessing the quality of products and practices that are 

selected for outreach. The criteria for reviewing educational products 

may be applied (1) according to rigorous standards of empirical precision, 

(2) according to informal judgmental estimates of product effects, or (3) 

anywhere along a continuum between the two. Decisions about the rigor of 

standards for reviewing educational innovations will have to be made in 

light of the situation existing in the local project site: 

° What resources for assisting in program evaluation are available 
to the project? 

° How valid and reliable are the existing product or practice 
evaluation data? 

° What kinds of evaluation data are considered important by 
potential adopters? 

° How much evidence of effectiveness is necessary to persuade 

potential adopters to pay attention to the innovation? < 

° What are the opportunity costs of investing in the kind of 
evaluation that will yield credible evidence about product 
effects? 



Informal 
Judgment 
of Product 
Effects 



Precise 
Empirical 
Evidence of 
Effectiveness 



Continuum of Standards for Assessing Educational Products 
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The standards employed in reviewing an innovation may be a'imed 
toward establishing claims of product effectiveness that could be 
submitted to the JDRP. This, as noted already, is the objective 
of the validation process and is described in detail in the next 
section"] The emphasis on empirical evidence of change that is 
typically accepted by the JDRP need not, however, be viewed as a 
limiting factor in decisions related to evaluation design. 

The outcome of careful, systematic product review--using this set 
of criteria and rating forms--may be evidence of product effective- 
ness that meets the standards for JDRP approval, but it need not be . 
The credibility of the evaluation of a Teacher Corps product or 
practice can be established without large amounts of empirical data. 
The effectiveness of some educational practices and processes cannot 
be established by conventional experimental procedures, data collec- 
tion, or statistical analyses. 



One approach to organizing project personnel for the review of project- 
developed products or practices is outlined below. 



Project Review 



A. The Roles 

The Applicant - The individual (or individuals) of a project who 
prepare the documentation for the review process for consideration 
by the Project Review Committee. 

The Project Review Committee - A group of three to five persons 
(including the project director and representing the LEA and IHE 
project staff and the Community Council) who review products and 
practices prior to dissemination to other Teacher Corps projects 
and/or other educational audiences. 



B. The Process 

A Local Teacher Corps project will establish a Project Review 
Committee to assess: 

^ The potential of products or practices (either under 
development or after implementation locally) for 
institutionalization and for use by other Teacher Corps 
projects, and/or 

^ The suitability of products or practices for dissemi- 
nation to educational agencies outside Teacher Corps. 
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Review Committees use the product rating form in this 
handbook or other suitable criteria to assess the 
potential of products or practices as: 

° Promising innovations (under development or already 
implemented) for dissemination through the variety 
of information exchange mechanisms that operate 
within Teacher Corps and beyond to other educational 
audiences. 

° Innovations considered suitable for external adoption 
or adaptation; information may be submitted as part 
of the validation process, described in the following 
section (pages 35 to 49). 



C. Guidelines 

The applicant will be allowed to withdraw a product or practice 
from the review process at any point during the procedure. 

The applicant will be responsible for preparing all descriptive 
material about the product or practice to be reviewed and will comply 
with the application procedures and criteria for submission to the 
dissemination system selected. 

D. Resources for Review and Validation 

Pages 16 to 33 of the handbook provide a rating instrument* that 
Teacher Corps project personnel may duplicate and adapt for use in 
reviewing educational products or practices. The appendices of the 
handbook contain other materials that may assist in the review process 
these include: 



Topic 


Handbook Section 


Pages 


Assessment 
criteria 


Appendix A. Operational 
Definitions of NIE Criteria 


51-58 


ERIC 

Criteria 


Appendix B. Criteria for 
Submission of Educational 
Materials to ERIC 


59-64 



♦Adapted from the NIE Product Rating Form (Susan Klein, 1974) 

23 
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The following materials may also be helpful in conducting project reviews 
and, as well, project validations: 



ED Materials/Support Center, National Diffusion Network, Far West Laboratory 
for Educational Research and Development, San Francisco, CA: 

*Guide to Packaging Your Educational Program , 1980. 

*Guide to Producing Print Materials , 1979, 



Educational Resources Information Center (ERIC), National Institute of 
Education, U.S. Department of Education, Washington, DC, 1979; 

*How to Use ERIC . 

^Submitting Documents to ERIC . 

Emrick, John A., with Peterson, Susan M. A Synthesis of Findings Across 
Five Recent Studies in Educational DTssemi nation and Change (Revised 
Version). Far West Laboratory for Educational Research and Develop- 
ment, San Francisco, CA, June 1978. 

EPIE Institute. Criteria for Reviewing Educational Products . Product I, 
NIE Products on Instructional Materials Selection (DHEW). ERIC 
Clearinghouse on Information Resources, Syracuse University, 
Syracuse, NY, 1978. 



Teacher Corps Dissemination Project, Far West Laboratory for Educational 
Research and Development, San Francisco, CA, 1980: 

Guidelines for Dissemination of Teacher Corps Products and Practices . 

*Manual for the Development of Educational Materials . 

Teacher Corps Dissemination Checklist . 

Teacher Corps Dissemination Sources . 

Teacher Corps Dissemination Tool Kit (includes publications listed 
here that are marked with an asterisk [*]). 



Technical Assist ice Base of the National Diffusion Network. Capla 
Associates, Inc., Rochelle Park, NJ: 

^Developer/Demonstrator Task Analysis . 

*Getting It All Together: The JDRP Process . 
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E. Product Rating Form 



This form was designed to facilitate the review of the quality of 
educational products for decisions related to continued product ' 
development, additional product evaluation, product dissemination 
or product utilization. The form is not designed to review technical 
plans for carrying out any of the above activities. This form has 
been developed to indicate dimensions of apparent quality of products 
even when highly credible evidence is not available to confirm these 
ratings. 



DIRECTIONS 



Sources of Information 

In determining your ratings, please consider all available evidence 
including the product itself (as well as samples of the product in use), 
qualitative or quantitative evidence provided by the producers (developers 
or publishers), judgmental or empirical information from other sources 
and information derived from your experiences and knowledge of the 
field. Feel free to contact developers, users, or others to secure 
additional information or to verify or clarify existing information. 



Ratings on Major Criteria 

After reviewing the relevant evidence on the prime and comparative 
products and determining the credibility of this evidence, circle the 
number 0 to 4 which reflects your best judgment of the value of the 
product on the relevant criteria dimensions. As indicated by the 
Product Review Form, the major criteria include: Desirability, Intrinsic 
Quality, Cost/Adoptability, and Effectiveness. These criteria dimensions 
are further subdivided to obtain more precise ratings .... On the 
ratings of each type of evidence, -please star (*) the considerations 
which primarily contributed to your judgments. Also, wherever possible, 
try to keep your ratings on the different criteria dimensions distinct 
so that the dimensions will assess unique factors. For example, the 
rating on the "desirability" of the product should not be altered by 
its rating on "cost" or "effectiveness." 

Overall ratings for each major criteria dimension and for the total 
product quality should reflect the average of the sub-ratings unless 
there is a defensible rationale for rating otherwise— such as a plan to 
weight the contributing criteria differently. 
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CRITERIA WEIGHTING FOR THE PRODUCT RATIMG FORM 



Teacher Corps Review Comnitees may use the Product Rating Form to assess 
Jl^ti"'!''^^ °^ ccmpleted products. PLEASE ASSIGN PERCENTAGE VALUE POINTS TO 
EACH OF THE CRITERIA. Sum these criteria tor each major grouping such as 
Desirabnrty, Intrinsic Quality, Cost and Adoptability, Effectiveness and 
Other. (Please use whole numbers 1-100.) Assume that the evidence to 
rate the criteria is available and credible. 



A. Dt5irability 
1. Need 

2. Marketability and Demand 

B. Intrinsic Quality 

1. Social Fairness 

a. sex 

. b. race 

c. rel igion 

d. socio-economic 

2. Content Accuracy 

3. Technical Quality of Communication 

4. Instructional Quality 

5. Product Appeal 

, 6. Uniqueness 

. 7. Adequacy of Product Development 

C. Costs and Adoptability 
. 1. Costs 

a. Reasonableness 

b. Compared to Common Alternatives 

c. Compared to Critical Competitors 



ERIC 



, 2. Adoptability 

a. Acceptability 

b . Expo rtab i 1 i ty 

c. Flexibility 

d. Ease of Use, Manageability 

D. Effectiveness 

1, Type and Comprehensiveness 

2. Potency (size, significance, duration) 

3. Extent of Initial Target and Non-Target 

Groups Effected 

4. Cumulative Nature of Effects on Non-Initial 

Users 

a. Snowball effect 

b. Impact effects 

5. Interactive Effects 

6. Efficiency 

E. Other Important Criteria (please specify) 



100% Total Points Possible 
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When used with developing Teacher Corps products or practices 
reviewers are encouraged to revise their ratings as new or 
more credible evidence emerges during the development ard 
evaluation phases. 

Bear in mind, however, that the instrument was designed for 
use by reviewers other than the developers and that the 
"closeness" of project personnel to an innovation will affect 
thetr objectivity < 



Credibility of the Information 

The ratings on the criteria dimensions should be influenced by the 
credibility of the evidence reviewed. If some of the sources of evidence 
or types of information have low credibility (0-1 ratings), do not base 
your judgments on this low credibility evidence. At the minimum, it is 
hoped that you will attribute at least passable credibility to your own 
analysis and judgments and thus be able to rate all the relevant dimension* 
requested in this Product Rating Form even if credible information from 
other empirical or judgmental sources is lacking. 

To help in later group discussions of the ratings and in feedback to the 
producers, please indicate by a check (/or //) which source of evidence 
most influenced your judgment. Then rate the credibility of all "sources 
on a scale from 4 (high) to 0 (low). 

Scale values associated with credibility or bel ievabil ity of the evidence 
include: 



4 — 



3 - 



Excellent Credibility — Complete confidence in the evidence 
because: the quality and appropriateness of evaluative 
measures and evaluation procedures are impeccable; the 
information is objective and generalizable on many relevant 
dimensions; and the effects referenced are supported by 
adequate evidence of causality. (These considerations are 
described in more detail in the chart on Credibility Dimensions 
Associated with Each Level of Evidence Presented.) 

- Good Credibility — Moderate confidence in the evidence 
because: the measures and evaluation procedures are appropriate 
and of sufficiently high quality; there is no reason to question 
the objectivity of the information, there is some indication 
of causality where an effect is claimed; and information on 
general izabil ity is limited. 

2 — Passable Credibility — Minimum confidence in the evidence 

because: although there may be better approaches, the measures 
and evaluation procedures are acceptable; while the data may be 
subjective, there is no conflicting information to question its 
validity; generalizability is limited; and evidence related to 
causality is weak. 

o 28 
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1 Doubtful Credibility Partial confidence in major portions 
of the evidence because: the measures and evaluation proce- 
dures are not completely acceptable; the information is more 
subjective than objective; the general izabi lity is severely 
limited; and there is little or no evidenc:. of causality. 

0 Negligible or No Credibility — • No confidence in major portions 
of the evidence because: the measures and evidence procedures 
are unacceptable; the information is subjective; the generali- 
zability is severely limited; and there is no adequate evidence 
of causality. 
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A. DESIRABILITY: NEED AND MARKETABILITY 



1. Need 

Assuming that this product is as ideal as it is described to be, use 
the following considerations in rating the need for the product in 
Its given domain: 



Social and educational significance of the problem (existing 
state in education and desired end state) 

—Size of population directly and indirectly affected 
by the problem 

—Intensity of the problem (demand for a solution) 

--Priority of the problem (importance to the decision 
maker-funding source) 

Relevance of product to general problem: Extent to which 
the product including side effect dimensions responds 
positively to the problem 

—Scope and centrality of the product to the need 

—Appropriateness or timeliness of the product for the need. 
Is there a need or a desirable want that is not already 
served? 

—Size of population which is likely to be served by the 
product or appropriate and options includirjg potential 
multiplicative or impact effects on secondary targets 
or beneficiaries. 



Negligible Extremely important 
need need 
0 1 2 3 4 

Indicate High (4) to Low (0) on credibility of the information 
on which you based your judgment: 

rationale of need provided by producer, other judgmental 

information, empirical information provided by producer, 
other empirical information. 
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2. Marketability and Demand 

Assuming that this product is as ideal as it is described to be, use 
the following considerations to rate the potential marketability or 
demand for the product: 



Availability of other products to meet the need 

Likelihood that product would be used if available (market 
study--for mass or thin market, publisher commitments) 



Probable Negligible 
Demand for the Product 
0 



Likely Strong Demand 

for Product 
4 



Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgment: 

information provided by producers, other judgmental information, 

other empirical information. 



Overall Rating on the Desirability of this Product Assuming that it is as 
Ideal as it is Described to be. (consider above two (2) ratings) ^ 



Negligible 
desirability 
0 



Extremely 
desirable 
4 



Please discuss your reasons, frame of reference, weighting for 
different types of evidence, etc. 



B. INTRINSIC QUALITIES (Items dealing with the essential nature or consti- 
tution of the product) 



1. Social Fairness - extent to which the product excludes and counters 

bias related to sex, race, religious beliefs, 
ethnicity and socio-economic status 



Contains obvious bias 
related to sex 

0 1. 



Affirmative Action appears 
to counter bias related to 
sex by appropriate corrective 
measures 
3 4 
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b. Affirmative Action appears 
Contains obvious bias to counter bias related to 
related to race and race and ethnicity by appro- 
ethnicity priate corrective measures 
0 1 —2 3 4 



Contains obvious bias 
related to religious 
bel i ef s 

0 



d. 



Contains obvious bias 
related to socio-eco- 
nomic status 

0 1- 



Affirmative Action appears 
to counter bias related to 
religious beliefs by appro- 
priate corrective measures 
■ 3 4 



Affirmative Action appears 
to counter bias related to 
socio-economic status by 
appropriate corrective 
measures 
-3 4 



Indicate High (4) to Low (0) on the credibility of the information or 
analysis on which the social fairness judgments are based. 

producer information, others' analysis of the product, 

your analysis of the product 



Content Accuracy and Validity - extent to which this product is accurate 

and appropriately complete; for learner 
level and up-to-date with current knowledge 



Contains serious flaws 
or inadequacies 

0 1- 



Content is accurate and 
appropriately complete 
for learner level 
4 



Indicate High (4) to Low (0) on the credibility of the information 
or analyses on which these content accuracy and validity judgments 
are based: 

producer information, others' analysis of the product^ 

your analysis of the product 
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Technical Quality 
of Communication 



extent to which audio and/or visual materials are 
presented, edited, synthesized, etc. to communicate 
clearly and appropriately 



Hampers communi- 
cation of content 
0 



1. 



Greatly facilitates 
communication of 
content 
4 



Indicate High (4) to Low (0) on the credibility of the information 
or analyses on which this technical quality is assessed: 



producer information, others^ 

your analysis of the product 



analyses of the product, 



Instructional Quality extent to which instructional strategies are 

appropriate for target groups 

Consider: 

® wise use of learning strategies, motivational techniques 
® correct reading level 

® inclusion of evaluation components as appropriate 
^ appropriate sequencing and pacing 

Apparent Highly appropriate 
serious strategies known to 
flaws work with similar users 
0 1 2 3 4 

Indicate High (4) to Low (0) on the credibility of the information 
or analysis on which this instructional quality is based: 

producer information, others* analysis of the product, 

your analysis of the product 



Product Appeal - extent to which product migiit appear artistic, creative, 
pleasing or satisfying to potential purchasers or users 



Product might appear 
unappealing, blah, 
preachy or condescending, 
disagreeable or unsatis- 
factory to majority of 
target users 

0 ^1- 



Product would appear 
to have great appeal 
and satisfaction for 
potential target users 
4 



Indicate High (4) to Low (0) on the credibility of the information 
or analysis on which this assessment of product appeal is based: 
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producer judgmental and empirical information, other judgments 

of the product, other empirical assessments of the product, your 

judgment of the product. 



6. Uniqueness or Innovativeness of Product in terms of: combinations 

of Intrinsic quality and other dimensions and 
methodological advances (models, etc., for im- 
proving education in other substantive areas). 



Commonplace, no Highly unique model 

different from for a new type of 

normal practice product 

0 1 2 3 4 



Indicate High (4) to Low (0) on the credibility of the information 
or analysis on which this assessment of product uniqueness is based: 

producer judgment and/or empirical evidence, other judgments 

of the product, other empirical assessments of the product, 

your judgment of the product. 



7. Adequacy of the Product Development Process - extent to which the product 

Is "well made." Consider the use of early task 
analysis, clear statements of purpose and objectives, 
appropriate relationships between tasks, appropriate 
formative evaluation and revision 



No information to 
suggest that the 
product was "well 
made" 

0 ' 1 2 



Product seems to 
have been carefully 
developed 



Indicate High (4) to Low (0) on the credibility of the information 
on which this assessment of the adequacy of the product development 
process is based. 

^ producer supplied background information, other background 

TfTFormation, your or others' assessment of"TFe product itself. 



Overall Rating on Intrinsic Quality 



Low Intrinsic High Intrinsic 
Quality Quality 
0 1 2 3 4 

f'lease discuss your reasons, weightings for different dimensions, etc. 
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C. COSTS AND ADOPTABILITY 



1. Costs 
Consider: 

^ materials, equipment, etc., provided for learners (distinguish 
between optional and required) 

^ materials, equipment, etc., provided for teachers and other 
personnel (e.g., orientation and pre- and inservice training 
materials and procedures, including description of the responsi- 
bilities of different individuals who contribute to the product's 
use). (Distinguish between optional and required.) 

^ personnel costs required for product adoption (number and types 
of personnel required, level of special training required, 
availability of special training, costs of training and/or 
consultants, and other necessary support services. 

® total product installation and continuation costs per user, 
school, or equivalent unit. 

^ cost per individual student or target recipient in terms of 
dollars and average hours of use required. 

a. Reasonableness of costs in dollars or dollar equivalent (see above 
considerations) for initial installation and continued use in terms 
of anticipated outcomes and economic reach of potential purchasers 
or users. 



Excessively high 
costs for poten- 
tial purchasers 
0 



-1. 



Low costs 
for potential 
pur chasers 
4 



b. Comparative rating of costs in relation to the costs for common 
alternatives.* 



Excessively high Low costs com- 
costs compared to pared to common 
common alternatives alternatives 
0 1 2 3 4 



♦Common alternatives are usual practices such as teacher lectures which are 
designed to achieve similar purposes. Critical competitors are exportable 
products designed to achieve similar purposes. 
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c. Comparative rating of costs In relation to critically competitive* 
products, if any exist. 



Excessively high 

costs compared to Low costs com- 
costs for critical pared to critical 
competitors competitors 
0 1 2 3 4 



Indicate High (4) to Low (0) on the credibility of the Information 
on which you based your judgments: 

cost information provided by the producer, externally 

supplied or verified cost data, external cost estimates. 



2. Adoptability 

a. Acceptability to purchasers and users 
Consider: 

° lack of opposition by target users or beneficiaries, 
by purchasers or others involved with the product 

° legality of use, conforming to appropriate copyright 
provisions, state laws, etc. 



Extensive user or 
purchaser opposition 
to product 

0 



Highly acceptable 
to users or 
purchasers 
4 



Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgments: 

information provided by the producer, other judgmental 

Information, other empirical information 



♦Common alternatives are usual practices such as teacher lectures which are 
designed to achieve similar purposes. Critical competitors are exportable 
products designed to achieve similar purposes. 
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b. Exportabnity 
Consider: 

* Is the product sufficiently complete to be used outside 
of the developmental sites? (I.e., on a "stand alone" 
basis without developers present) 

^ are the product and Its supporting materials amenable 
to adequate dissemination? 

^ If auxiliary services such as teacher training, equipment 
maintenance, or external testing services are needed, to 
what extent are they likely to be available in user sites? 



Very difficult to 
use outside the 
development setting 
0 



Easy to use out- 
side the develop- 
ment setting 
4 



Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgments: 

information provided by producer, other judgmental 

information, other empirical Information. 



c. Flexibility - to what extent can the product be easily changed 

or adapted to local needs or used to fit multiple 
needs and constraints? 



Product Adaption Product Adaption 
not recommended or recommended and 
quite difficult quite easy 
0 1 2 3 4 



Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgments: 

information provided by the producer, other judgmental 

information, other empirical information. 



d. Ease of Use: Manageability 
Consider: 



o 



consumer readiness: degree to which product can 
be used under pre-existing user constraints 

^ extent of resource requirements (equipment, personnel, 
space) not normally available in schools 
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^ extent of organizational changes needed In scheduling, 
organizing personnel, revising management systems, etc., 
to fit with existing components of the educational 
system 

^ ease of consumer maintenance, product durability (or ease 
of updating products), usefulness of auxiliary materials 
(teacher guides, evaluation procedures) quality control 
mechanisms 



Not easily managed 
with common existing 
resources 

0 1- 



Easy to operate, 
use with common 
existing resources 
' 4 



Indicate High (4) to Low (0) on the credibility of the Information 
on which you based your judgments: 

information provided by the producer, other judgmental 

Information, other empirical information. 



Overall Rating on Cost and Adoptability 



High cost, difficult 
to adopt or adapt 

0 



Low cost, easy to 
adopt or adapt 
4 



D. EXTRINSIC EFFECTS—EFFECTIVENESS 



!• Type and Comprehensiveness of Learner Effects on Appropriate Target 
Population 

Consider number of positively affected key variables which are related 
to: 

^ learner knowledge, attitude, or awareness changes (as 

Indicated by paper and pencil assessments or the equivalent) 

^ learner performance or skill changes (behavioral indicators 
such as application tests, simulations, or 'Veal life" ob- 
servations over short and long term periods). 
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Few trivial types of Many Important types 

positive learner of positive learner 

effects effects 

0 1 2 3 4 



Indicate High (4) to Low (0) on the credibility of the Information 
on which you based your Judgment: 

empirical evidence provided by the producer, judgmental 

evidence provided by others (own), straight empTrlcal evidence 

provided by others, comparative empirical evidence. 

(Evidence supporting the credibility Issue of causality 
would have to be examined carefully. Performance 
changes would probably be weighted more highly than 
cognitive changes.) 

Potency of the Effects 
Consider: 

^ size and significance of the change resulting from the 
product on the prime target group 

^ duration of the ef.'oct 

(The credibility of measures and evaluation 
procedures would have to be examined carefully). 

Extensive, signi- 

Small, insignificant f leant long-term 
short-term effects effects 
0 1 2 3 4 



Indicate High (^) to Low (0) on the credibility of the information 
on which you based your judgment: 

^ empirical evidence provided by the producer, judgmental 

evidence provided by others, straignt empirical evidence 

provided by others, comparative empirical evidence. 

(Evidence supporting the credibility issue of causality 
would have to be examined carefully. Performance 
changes would probably be we" Hted more highly than 
cognitive changes.) 
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3. Extent of Initial Target and Non«Target Groups Affected Positively 
Consider: 

• extent of positive effects (minus negative effeeti) on 
the target group (intended recipients to solve educa- 
tional problem) 

* extent of positive effects minus negative effects on the 
non-target group (unintended recipients) 



Positive effects on a Positive effects on 
limited segment of the total target group 
target group and on few and many non-target 
non-target 9*oup members group users 
0 1 2 J 4 

Indicate High (4) to Low (0) on the credibility of the information 
f.ri which you based your Judgment: 

empirical evidence provided by the producer. Judgmental 

evidence provided by others, straight empiricaPevidence 

provided by others, comparative empirical evidence. 

(The credibility issue related to the extent of generali- 
zability to target and non-target populations should be 
examined carefully. Positive effects on target groups 
would probably be weighted higher than positive effects 
on non-target groups.) 



4. Cumulative Nature of the Effects on Others (not the original users) 
Consider: 

a. Multiplicative or "snowball" effects (e.g., original 
users will be able to train other users to do the same 



thing) 



Negligible "snowball 
effects 

0 



II 



1 



2 



3 



Extensive "snowball" 

effects 

4 



b. Impact effects (e.g., original users such as teachers will 
learn to behave in ways that have positive results on their 
students) 



Negligible positive 
impact effects on 
individuals who are 
original users 



Extensive positive 
impact eflects on 
Individuals who are 
original users 



0 



1 



2 



3 



4 
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Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgment: 

^ empirical evidence provided by the producer, judgmental 

evidence provided by others, straight empirical evidence 

provided by others, comparative empirical evidence. 

(Evidence supporting the credibility issue of causality 
would have to be examined carefully. Performance changes 
would probably be weighted more highly than cognitive 
changes.) 



5. Interactive Effects of the Product (treatment) with other products 
(treatments) under conditions, or user characteristics. 
Do the effects on t he original users become stronger o r 
v^eiafk'ernwhSTrTfs^ with orner treatments or after specified 
pre-requisite treatments? Many interactive effects may 
be labeled as positive or negative "side effects." 



Negative interactive 
effects of the product 
and other products or 
user conditions or 
characteristics 

0 . 



Positive interactive 
effects of the product 
and other products or 
user conditions or 
characteristics 
1 — 4 



Indicate High (4) to Low (0) on the credibility of the information 
on which you based your judgment: 

^ empirical evidence provided by the producer, judgmental 

evidence provided by others, straight empirical evidence 

provided by others, comparative empirical evidence 

(Credibility issues related to general izabi lity and 
external validity should be examined carefully.) 

6. Efficiency in Achieving the Effects 
Consider: 

° amount of learner time required to attain effects 

° amount of other (non-dollar) resources required to attain 
effects (e.g., numbers of instructional personnel required) 



Negligible efficiency 
in achieving effects 
0 



Extensive efficiency 
in achieving effects 
4 
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Indicate High (4) to Low (0) on the credibility of the information 
on which you ba..ed your judgment: 

empirical evidence provided by the producer, judgmental 

evidence provided by others, straight empirical evidence 

by others, comparative empirical evidence. 

(Evidence supporting the credibility issue of causality 
would have to be examined carefully. Performance changes 
would probably be weighted more highly than cognitive 
changes.) 



Overall Rating on Effectiveness of Product 

Negligible positive 
effects — 



Extensive positive 
effge^ts 



Please discuss your reasons, weightings for different dimensions, etc. 



OVERALL RATING ON TOTAL PRODUCT QUALITY 



Inferior 
quality 
0 



Excel lent 
quality 
4 



Differential weightings of the major types of evidence should be 
explained. 
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GUIDELINES FOR PROJECT VALIDATION 
OF INNOVATIVE PRODUCTS AIID PRACTICES 
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External Quality Standards 

The following guidelines suggest procedures for Teacher Corps project 
validation of educational products and practices for submission to the 
Joint Dissemination Review Panel (JDRP). Similar procedures are prescribed 
for many state IVD processes. Each project will decide on the specific 
processes for validation, including administrative procedures, allocation 
of project funds, documentation requirements, and criteria in addition to 
those of the JDRP or the state IVD review. 



Though projects may decide on more or less rigorous requirements for 
preliminary screening of innovations, the investigation of claims of 
effectiveness will focus on the submission materials required for review 
by the JDRP (as outlined on pages 41 to 48 of this handbook). 



Project Validation 
A. The Roles 

The Applicant - The director of the Teacher Corps project applying 
for validation of a project-developed innovation. 

Investigator or Investigation Team - Two options are available: 

° The Advocate or Advocate Team - A Teacher Corps project director 
(possibly teamed with one other resource person selected by 
the applicant) who may be designated to conduct the investigation 
of the product or practice. 

° The Validation Agent - An expert (not associated with Teacher 
Corps) who may conduct the investigation of the claims of 
effectiveness for the innovation. 

Validation Committee - A committee of three to five members, . 
representing project directors, deans, and the LEAs and IHEs 
of other projects in the state or region.* The committee will be 
responsible for reviewing the recommendations to the Teacher Corps 
program office. It may also provide subcommittees to investigate 
project innovations. 



*Obviously, this may not be feasible for projects located in some areas of 
the country. A committee of experienced educators could be formed with 
members from other components of the IHE and LEA. 
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The Teacher Corps Washington Dissemination Unit - The person 
designated by the program office to submit the transmittal 
memorandum and each submission that has completed the 
Teacher Corps project validation process to the Joint 
Dissemination Review Panel. 

B. The Process 

Advance Information Sharing - Local projects will prepare a 
brief prospectus for each product or practice they believe has 
potential for validation. These may be shared with nearby Teacher 
Corps projects in advance of the preparation of formal validation 
documents. A prospectus on an innovation may draw upon data 
accumulated during the local project review. Participants in each 
of these sharing situations can identify products and processes that 
appear to have sufficient evidence of effectiveness to warrant an 
application for validation. Each developer will be free to prepare 
formal application materials for JDRP review of the innovation. 

Initiation - The applicant will prepare a draft version of the 
Form for Submitting Materials to the Joint Dissemination Review 
Panel for review by the Validation Committee. The applicant will 
distribute copies to each member of the committee and, after a 
reasonable period (10-15 days), will poll the committee to 
determine if there is sufficient consensus among the members 
to continue the validation process. 

Validation Committee Action - The members of the committee will : 
(a) consider the application and recommend that it receive 

continuing val Idation _acjipi]i„p^^ (bLxecprnf^end that the application 

be rejected. If a majority of the committee rejects the initial 
request or causes the process to stop at any point short of 
successful completion, the committee chairperson will send a 
letter to the applicant giving reasons for such action. The 
applicant may respond in writing to each member of the committee 
within two weeks. The committee members will then reconsider 
their decisions and the chairperson will notify the applicant 
In writing of the committee's decision^ If the committee accepts 
the request, the applicant will be advised to schedule an 
Investigation. 

Procedural Options for Investigation - Teacher Corps projects may 
choose from the following alternatives In establishing the formal 
process for investigating the claims of effectiveness for project- 
developed Innovations: 

° Advocate (or Advocate Team) Investigation by a Teacher 
Corps project director or other qualified person selected 
by the Validation Committee. 

° Validation Agent investigation by a specialist in educa- 
tional evaluation selected by the Validation Committee. 
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Validation Committee investigation by a subcommittee 
selected from the Validation Committee. 

^ Combinations of the three preceding options may also 
be selected to expedite the validation process. 



The Investigation - Whichever of the three processes (or 
combinations of processes) for assessing project innovations 
is selected, a site visit provides an excellent oppportunity 
to determine objectively the accuracy of the claims of 
effectiveness made by the applicant. During a two-day visit 
the investigator (or investigation team) may include one day 
of pertinent interviews, review of evaluation data, and any 
other activity which might serve to familiarize the team 
with the innovation. Fhe second day of the visit would allow 
for completion of data gathering, analysis, preparation of a 
report, and opportunity to discuss th^ investigator's 
recommendations to the committee with che applicant. A letter 
report of the investigation should be sent to members of the 
Validation Committee within two weeks of the visit with one of 
the following recommendations: 

^ The Validation Committee should take no further action 
on the validation of the innovation; the investigator 
does not support validation of the innovation.* 

° The Validation Committee should take no further action 
until such time that the applicant brings evidence to 
the committee that specified developmental activities 
(for example, an additional field test) have been 
carried out in regard to the product or practice. At 
such a time, the investigator would support validation 
of the innovation. 

The Validation Committee should support validation 
of the innovation. 

In the event that the applicant and the investigator cannot 
reach mutual agreement on recommendations to the committee, the 
applicant may file an appeal to the committee within two weeks 
of the original investigation requesting a second investigation. 



Validation Committee Action - The investigator will send copies of 
the report to each member of the Validation Committee and to the 
applicant, and schedule a meeting (or conference call if that is more 
practical) of the committee. The committee will hear any final 
arguments for or against validation from* the applicant and/or the 
investigator. The Validation Committee will make its recommendations 
on validation using normal decision making rules. In cases of 
approval, the Validation Committee will direct the applicant to 
submit the validation documents to the Teacher Corps Washington 
Dissemination Unit for program review prior to submission to the JDRP. 
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C* Guidelines 

The applicant will be allowed to withdraw an innovation from 
consideration at any point in the validation process. 

The applicant will be responsible for preparing all descriptive 
material in regard to the innovative product or practice. 

The selection of an advocate or advocate team by the Validation 
Committee may take into account the following factors: 

Preferences of the applicant, especially for a 
resource person to serve with a project director. 

^ Equitable distribution of the advocacy workloa d 

among the other project directors. 

^ The individual director's desire to serve as 
advocate for a given validation effort. 

^ Background or experience in the technical areas 
which are related to the innovation to be vali- 
dated. 

If no director is willing to serve as an advocate for a given 
validation request, the Validation Committee will select an 
impartial investigation group from its own members or other 
available sources. 

The selection of the Validation Agent by the Validation Committee 
may involve consideration of the following factors: 

^ Qualifications in the technical areas related to 
the innovation to be validated. Acceptable evidence 
might be publications, research, reports, and/or 
recommendations of persons in responsible educational 
positions. 

Availability in terms of scheduling and fees that 
are not beyond project budget limitations. 

^ Whether or not the agent has had any involvement in 
the development of the innovation to be validated. 

Whether or not the agent is an employee of the in- 
stitution of higher education or local educational 
agency which sponsors the applicant's Teacher Corps 
project. 
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The selection of the Validation Committee may be done as follows: 

° Project directors will share with nearby projects 
the name(s) and background(s) of project directors, 
deans, and other project staff personnel of the 
SEAs and LEAs with expertise and interest in 
serving on validation committees. 

° Applicants will consult the roster of candidates 
(and accompanying background data) and invite in- 
dividuals to serve on a Validation Committee to 
review particular products or practices. 

The validation documents submitted to the Teacher Corps Washington 
Dissemination Unit will include the completed form for submitting 

materials to uDRP, and may include the Investigation Report and 

other documentation of the Validation Committee. 

D. Resources for Validation 

The appendices contain information that may be useful in conducting 
a project validation: 



Topic Handbook Section Pages 

Criteria Appendix A . Operational 51-58 

Definitions of NIE Criteria 

Format Appendix C . Specimen JDRP 65-75 

Submission 



The JDRP and IVP process are described in detail in the followi 



ng: 



Tallmadge, G. Kasten. The Joint Dissemination Review Panel Ideabook . 
U. S. Department of Health, Education and Welfare: Office of 
Exiucation and the National Institute of Education, October 1977. 
Available from ERIC, ED 148 329, or from the Superintendent of 
Documents, Government Printing Office (GPO), Washington, DC 20202, 
Stock #017-080-01824-1; $3.00 prepaid). 

Hi nze, William. Sharing Educational Success: A Handbook for Validation 
of Educational Practices . (Revised Edition, GPO 861 549). (Copies 
may be requested from Dr. William H. Hinze, National Diffusion 
Network, Room 631, Transpoint Building, Washington, 20202). 

The documentation of a Teacher Corps project's experience with the JDRP is in: 

Nagel , Thomas. A Teacher Corps Case Study: The National Validation of 
SIGMA. Teacher Corps Developmental Training Activities, University 
Nebraska at Omaha, 1980. 

See also the list of publications on page 15 (above) for additional materials. 
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E. JDRP Format 



FORMAT AND INSTRUCTIONS FOR SUBMITTING MATERIALS* 
TO THE JOINT DISSEMINATION REVIEW PANEL 



The outline on the following pages displays the format 
that should be used for submitting materials to the Joint 
Dissemination Review Panel. Under each major heading are 
specifications for the kinds of information that should be 
included. 

--I-n-opdep-to-be - co rwHie4ftgT-^-hfr--eva^htat i on ■ need not tje 

a strict experimental design, although this type of evidence 
is desirable. However, there should be some kind of high- 
quality, objective, methodologically sound, quantitative 
evidence demonstrating that the intervention in question was 
effective and superior to other, more commonly used methods or 
approaches, and that the observed effects were caused by the 
intervention. 

The total length of each submission, including all de- 
scriptive material, tables, etc., should not exceed ten pages. 
Ten pages is a maximum; less lengthy presentations are definitely 
acceptable and encouraged. The evidence needs to be convincing, 
not necessarily lengthy. Some of the strongest arguments are 
brief, concise, and to the point. 

While brevity is a virtue, the materials should include all 
the information the panel will need to make its decision. All 
submissions should follow the format by using the headings shown, 
and should cover all the points mentioned in the instructions 
under each heading. The points under each heading should be pre- 
sented in the order most convenient and logical for the intervention 
being documented, not necessarily in the same order as in these 
i nstructions. 



* Ideabook , Tallmadge, 1977. 
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FORMAT FOR SUBMITTING MATERIALS 
TO THE JOINT DISSEMINATION REVIEW PANEL 

I 

PROGRAM AREA: (e.g., Title III, reading, career education, 
environmental education, education for the handicapped) 



I. INTERVENTION TITLE, LOCATION: 

Specify the title of the intervention and the 
location for which evidence of effectiveness is 
being submitted. 

II* DEVELOPED BY: 



Indicate who developed the intervention originally, 
even if this happened at a different site than the 
one for which evidence of effectiveness is being 
presented. 

III. SOURCE AND LEVEL OF FUNDING: 

List all funding sources for the intervention at the 
location for which evidence of effectiveness is pre- 
sented and, for each source, list the amount of funds 
(see rigure 1 for an example). 

IV. YEARS OF INTERVENTION DEVELOPMENT: 

Indicate the year or years during which the inter- 
vention was originally developed or tested. 

V. BRIEF DESCRIPTION OF INTERVENTION: 

Briefly describe the intervention for which claims 
of effectiveness are being made. The description 
should cover at least the following points:" 

What is the intervention? 

What are its objectives? 

What claims of effectiveness are being made? 

What is the context in which it operates? 

Who are the intended users and beneficiaries? 

What are the characteristics of the groups on which 

the intervention was developed and tested? 
What are the salient features of the intervention? 
What are the costs for adoption and maintenance of 

the intervention? 
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EVIDENCE OF EFFECTIVENESS: 

Describe the evidence of effecti vaness for the Inter- 
vention. This section should deal with each of the 
following points, although not necessarily In the same 
order: 

Interpretablllty of measures : Evidence that the 
quantitative measures are reliable and valid Indi- 
cators of the effects claimed. 

Credibility of evidence : Who collected and analyzed 
the data, what assurances are there that the findings 
are objective? 

Evidence of Impact : What Is the evidence that some- 

thlngJiap^enPfl? ffhat are the effects cla4me<l^p^e 

Intervention? 

Evidence of statistical reliability of the effects : 
What Is the evidence that the effects happened often 
enough and with sufficient reliability to be likely 
to happen again under similar circumstances? 

Evidence that the effects are educationally meaning ful : 
what is the evidence that the effects are large enough, 
powerful enough j or Important enough to be educationally 
meaningful, regardless of their statistical significance? 

Evidence that the effects are attributable to the Inter- - 
vention : Can alternative explanations such as practice 
effects, maturation, selection of superior treatment 
groups, etc., be ruled out? 

Evidence of generallzablllty to the populations for which 
the product or practice is intended : Evidence that the 
product or practice has been tested widely enough and 
under sufficiently diverse circumstances to give assurance 
that the effects claimed may be similar when the product 
or practice is used elsewhere for the populations intended. 
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I 





Source and Level of Funding of Intervention 




INSTALLATION SUBSEQUENT YEARS 
(Non-recurr i ng (Recurr i ng 
Costs Costs) 


Personnel 

Personnel Training 
Equipment & Materials 




Consumables 

Other Costs* - Specify: 
TOTAL 


• 

1 


* E.g., transportation, technical assistance, public 
relations, etc. 



Figure 1. An illustration of a table shell for 
showing costs. 



i 
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JDRP Criteria for Judging Effectiveness 

The Joint Dissemination Review Panel has received requests for 
a statement of the "cutting points" on the criteria used in 
reviewing projects. The Panel has been unwilling to set forth 
precise specifications. The decision as to whether the evidence 
of effectiveness on any particular product or practice justifies 
use of Federal funds to promote dissemination is not automatic 
or mechanical. This decision involves judgment about the educa- 
tional signifcance of the claimed effect and the credibility of 
the evidence supporting the claimed results. A decision that 
the evidence is not convincing is usually not due to a single 
fatal flaw, but to a failure of plausibility on several criteria. 

The Panel believes that setting up highly specific standards 
relating to educational significance (e.g., the importance, 

number, magnitude, durability, and consistency of positive 

ettects and their comparative advantage) and credibility of the 
supporting evidence (e.g., objectivity, reliability, validity, 
generailzabi lity and causal nature of the data) would degrade 
rather than improve the review process. The Panel has felt, for 
example, that it would be irresponsible to assert that projects 
achieving a gain of 1.0 grade equivalent units should be accepted 
while a gain of 0.9 grade equivalent units should be rejected; or 
that a sample size of, say, 50 is acceptable, whereas a sample 
size of 49 is not. 

It is because a complex judgment on plausibility is required—one 
that takes into account all of the evidence— that the Joint 
Dissemination Review Panel is needed. Past experience indicates 
that leaving the judgment of what is acceptable up to each Educa- 
tion Division program produces variable and questionable results. 
On the other hand, developing a manual of boiler-plate standards 
would entail the arbitrariness cited above. 

Some Guidance 

All this may be true, but acknowledging it may not provide the 
guidance which Education Division program staffs need in order 
to identify products or practices which have a high likelihood 
of being approved. 

The purpose of this section is to set forth in a general way, 
but in greater detail than is furnished in the Form for Submitting 
Materials in the Joint Dissemination Review Panel, the kinds of 
criteria the Panel uses in assessing the effectiveness of the 
products or practices submitted to it. This should provide program 
staffs with better guidance for selecting potential exemplary 
materials and methods, and for preparing submissions to the Panel. 
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Many of the products and practices submitted to the Panel 
have been in the area of compensatory education, environmencal 
education, reading, mathematics, bilingual education, and other 
similar areas of cognitive achievement or affective gain in 
individual elementary and secondary students. The criteria 
discussed below as illustrative relate mainly to these kinds of 
submission. 

Perhaps the best way of making clear the kinds of cr'^'^^eria the 
Panel uses in reaching its judgments on effect iver.uss is to 
set forth a series of general, common-sense questions. The 
questions focus on three issues: (1) Is there evidence that 
anything important happened that is consistent with the stated 
claims? (2) Is there evidence that what happened is general i- 
zable? and (3) Can this credibly be attributed to the product 
or practice? 

"In — Ho\if^eH-^d-the'-learTiers-perforTTH)efo 

to the product or practice in question, and how well did 
they perform after experiencing it? As an example, are 
there measures taken before and after the learners experi- 
enced the innovation which show that they gained or improved? 

2. Are the gains statistically reliable? 

3. If there is evidence of change or improvement, is the gain 
large enough to be meaningful? That is, is it large enough 
to be both statistically reliable and educationally signi- 
ficant? Many submissions show apparent before-after mean 
changes in the expected direction, but they are either not 
statistically reliable or, even if statistically reliable, 
are too small in size to constitute meaningful educational 
improvement judged against commonly available experience. 

4. Is there evidence that the improvements registered by the 
product or practice are stable and general izable to other 
scRcrol"**^settT^^^ does it appear that they may be the 
result of special characteristics of the particular group 

of students who happened to use the product or practice when 
it was evaluated, or the unusual competence of the teacher 
who used it? These considerations are acute when the 
number of learners is small or where there is only a single, 
use of the product, i.e., where the evidence is based on 
one teacher and one classroom. Thus, it Is desirable, when 
the number of cases is small, for there to have been multiple 
replications of the practice (e.g., different classrooms, 
different teachers, different schools, or different years). 
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5. Once it has been established that there have been gains, 
that they are large enough to be statistically and educa- 
tionally meaningful, and that they are generalizable beyond 
the originating site, the central question remains: Can 
we be reasonably confident that the gains can be attributed 
to the innovation described in the normal maturation, to 
the regular education process, or to other special factors 
which may be affecting the students? In order to provide 
some kind of answer to this crucially important question, 
submissions to the Panel should include some measure or 
estimate of what would have happened to the students if 
they had not been using the product or practice. One of 
the best ways to do this usually, is through a control 
group, but the Panel has approved many projects which did 
not include a control group because the evidence of effec- 
tiveness contained some other persuasive measure or estimate 
of the effects on non-treatment. For example, comparisons 
of recent gains with previous year's scores or national 
norms often provide persuasive evidence that unusually 
large gains recorded can reasonably be attributed to the 
product or practice. Another example is time series com- 
parisons against a baseline time period. In many cases, 
however, simple before-after data on the user groups are 
likely to be ambiguous when there are no parallel data on 
a comparable, untreated control group. 

The idea is to provide evidence ruling out rival hypotheses 
(factors which may be responsible for producing the effects 
attributed to the product or practice). Examples of rival 
hypotheses are: 

* maturation--psychological or physiological changes 
in the users. 

* history— events or administrative procedures which 
occurred in addition to the product or practice. 

* selection/differential attrition— when comparison 
groups are unequal on the relevant measures at the 
beginning (du2 to selection) or at the end (due to 
differential attrition). 

* statistical regression— e.g. , when groups which 
are selected for their initial extreme scores on 
a given measure fall closer to their population 
mean score on retesting with the same measure. 

* test effects— increases in test scores due to the 
learning that results from frequent use of the same 
test, or to teaching to the test. ----v 
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These are the main criteria by which the Panel tries to 
assess the evidence of effectiveness presented for products 
or practices. It is not possible to specify the precise point 
on any of these criteria which categorically differentiates 
acceptable from un-acceptable submissions. Using all these 
criteria and examining all the evidence, the Panel tries to render 
judgments which, on one hand, will screen out those submissions 
whose evidence of effectiveness is not strong enough to support 
a federally funded dissemination effort, and on the other, will 
insure that those products and practices where there is such 
evidence, get moved into a wider process of dissemination which 
ultimately can improve education. 

Other Kinds of Outcomes 

It was noted above that these illustrations apply most appropri- 
ately to innovations directed at improving the achievement of 
elementary school and secondary school children. The illustrations 
discussed above would in many cases be quite inappropriate. The 
Education Division carries out a much wider variety of educational 
activities. Some activities seek to change attitudes or impart 
information. Others are aimed at bringing about changes in insti- 
tutional procedures, at persuading state and- local governments 
to adopt certain techniques, or at altering the ways in which state 
or local monies are spent. Still others seek to expand the pool 
of competent educational practitioners through innovations in 
inservice or inservice training. 

In all these cases, quite different outcome measures and evidence 
of effectiveness will be required. However, the same general logic 
implicit in the questions listrl above apply to these products and 
practices as well: Is there persuasive, objective evidence that 
things are different after using the product or practice than they 
were before? And can we be reasonably confident that any such 
changes or improvements noted can be attributed to the effects of 
the innovation described rather than to something else? 

Prereview Concern s 

JDRP's primary responsibility is to judge the effectiveness of 
submitted products or practices. But prereviews before presenta- 
tions to the Panel also serve as an additional opportunity to 
assure that other screening criteria which are the responsibili- 
ties of the Education Division's programs are met. As discussed 
in the first two sections of this paper, the Panel Is concerned . 
that the products or practices whose effectiveness is to be 
reviewed are clearly identified, that common implementation 
problems are described, and that full start-up and operating 
costs are presented so that schools interested in selecting the 
innovation will do so without misunderstanding its costs. Other 
federal concerns include social fairness and possible harm to 
users. All of these must prereviewed by program offices. 
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Many other concerns for "good" products have been suggested. 
These include the conceptual soundness of the product or 
practice, the quality of the developmental process, acceptance 
by educators, the uniqueness of the submissions, how profes- 
sional the materials look, and enthusiasm of parents and 
students. 

The Panel regards these considerations as important. Program 
offices are encouraged to prereview project submissions to 
the Panel on all such criteria. Nonetheless, the Panel's sole 
function is to review only one category of criteria that may 
be considered before dissemination: evidence of effectiveness. 
This may be considered a fairly large task by itself. Adopters 
should recognize, however, that JDRP only decides whether 
persuasive evidence of effectiveness consistent with the claims 
and goals has been presented. The product does not have a 
universal seal of approval" from the federal government: this 
is not the Panel's function. 
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APPENDIX A 



OPERATIONAL DEFINITIONS OF NIE CRITERIA 
FOR REVIEWING EDUCATIONAL PRODUCTS 
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Operational Definitions of Criteria* 

The term '^criteria" has been used in the study to denote the standards, 
guides, rules and issues of concern of decision-makers used in the process 
of reviewing and selecting instructional products. Operational definitions 
of criteria should identify the connotations of the term criteria, i.e., the 
parameters, usages and functions. It should be possible to determine what 
is and is not included in a criterion on the basis of its operational defi- 
nition, and thus, to make a decision as to whether that criterion is or is 
not met by the product, 

The constraints noted in a previous section of the chapter were in- 
tended to preclude the possibility of operationally defining criteria in 
such a way as to ensure theur general acceptability. The criteria can be 
defined only as they are used in the product review instrument of this study. 



Desirabi lity 



Need 

Schermer (1975, p, 4) defines "need" for a product as ", , , the 
social and educational significance of the , , , product, the size of the 
target group, and the priorities of the reviewing agency," This definition 
is applicable in the present study. 



Demand 

Demand will mean that there is evidence that the product was developed 
as a result of consumer-identified (as opposed to sponsor/developer-identi- 
fied) need and request. In reviewing the product, the scope and universality 
of the product should be considered. 

Demand will also mean that the product has high marketability as evi- 
denced by the dissemination and diffusion of the product. Consumers are 
willing to purchase the product. 



Appropriateness to Users 

Appropriateness for the user is the degree to which the product meets 
the needs of learners/users in terms of the product's instructional intent, 
its match with the age, grade and ability level of the target group, how 
well it integrates with the on-going instructional program, and the extent 
of its compatibility with the general educational objectives of the local 
setting. 



* Clark, Carolyn, Report on a Codification and Analysis of Assurances , 
Claims and Evidence to Support Claims in NIE Sponsored Educational 
Products , 1975, U.S, Department of Health, Education and Welfare: 
National Institute of Education, October, 1976. 
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Appropriateness to Coinmunity 

Appropriateness for the community is the degree to which the product 
is compatible with the instructional, social, moral and spiritual values of 
the local community. Issues such as citizenship, patriotism, respect for 
the free enterprise system, respect for authority, etc., may be considered 
among these values. 

In the desirability cluster, two of the criteria--need for the product 
and appropriateness to users--seem essential and may outweigh issues related 
to appropriateness to the community if, should controversy arise, it can be 
shown that need and user appropriateness are met by the product better than 
anything else available on the market. Demand, as defined here, is least 
essential and can only be valued in conjunction with at least one other 
criterion. 



Practical ity 



Cost 

Cost of the product must be judged on the basis of available resources 
of the decision-maker, the relationship between cost and effectiveness of 
the product, and the advantages of the product when compared with similar 
other products. Costs are interrelated with other criteria such as training, 
availability, durability and replicabil ity of the product. Costs depend 
also upon per student use. 



Ease of Use 

Ease of use is evaluated in terms of the clarity and specificity of 
tasks associated with implementation of the product for teachers and 
learners; the interrelationship of the product with other learning and in- 
structional tasks and programs; and the satisfaction realized by users of 
the product under normal instructional conditions. 



Training Requirements 

Consideration of the training requirements of the product should in- 
clude the cost in personnel time and availability, the relationship of the 
need for training to the quality of implementation, and the independence 
of the training program from the developer or other external agents. 



Preparation Time 

Preparation time, i.e., time required outside the classroom or other 
instructional setting, for grading, setting up activities, managing equip- 
ment, etc., should be minimal, and should not detract from time scheduled 
for other prep-time responsibilities of the implementer. 
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Availability 

Availability of the n.aterials, equipment or training necessary for 
implementation of the product must be assessed in terms of the ability of 
developers or publisher to meet user delivery deadlines, to replace or 
repair failing materials or equipment, and to adhere to agreed upon costs 
and quality of the product. 



Durability 

Durability of the product is defined by Coller (1976) as the extent to 
which the product has continuance, in the sense of not soon becoming obso- 
lete, sturdiness in the sense of enduring hard use, and perpetuality in the 
sense of being easy to update so that it has lasting utility. 



Support Staff 

The need for support staff, such as clerks or paraprofessionals, has 
implications for cost, training and ease of use of the product, and should 
be assessed in terms of current staff resources and the relationship of 
staffing to implementability of the product. 



Adaptability 

Adaptability is defined by Coller (1976) as the extent to which the 
product can be easily changed to fit better the institutional conditions 
of the consumer without losing any of its essential qualities and/or 
effectiveness. 



Replicability 

Replicabil ity is the extent to which the user must adhere to strict 
implementation standards in order to achieve the intended outcomes of the 
product; and the extent to which it is possible to achieve those outcomes 
without the participation or intervention of the developer or other external 
agent. 

In the practicality cluster, the most essential criterion is the cost 
of the product and concomitant issues of training, durability, availability 
and adaptability. 



Social Fairness 



For purposes of the study, the social fairness product review criteria 
relate to the qualities of the product in terms of fairness, balance and/or 
affirmative treatment of women and minorities. These features of the 
product can best be assessed by careful review of the product itself, though 
information about the product can provide basic screening guidelines. 
Characteristics to be reviewed are the use of language when referring to 
races, nationalities, religions, the sexes and so forth. 
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Intrinsic Qualities 



Choice and Accuracy of Content 

Content issues require careful product examination. Content issues 
include issues previously addressed under desirability, such as the ap- 
propriateness of the materials for the target group's age, grade and 
ability levels. The reviewer is concer.ied about the breadth of the con- 
tent in terms of its currency. Content issues include the treatment of 
current social, technological and scientific developments as they relate 
to the specific subject matter. 



Instructional Methods 

The instructional method should account for individual learner 
differences as well as be appropriate to the content. The method of in- 
struction should allow easy implementation as well as adaptation. The 
scope and sequence of the materials should be clear, as should be the 
learning tasks. Evaluation procedures to be used by students should be 
clearly defined. 



Educational Principles/Philosophy /Value Orientation of the Product 

This criterion is one •wlTich may be of more interest to the community 
than to the students. It is an important criterion when screening or 
reviewing products for classroom use. For purposes of the study, the 
criterion will be documented rather than evaluated. 



Clarity of Objectives 

It Is useful to have clearly stated instructional objectives from 
the developer/producer. The task of determining the appropriateness of 
the product to the setting is aided when the objectives are clearly stated. 
The objectives of the product will related to choice of content and appro- 
priateness of the instructional design. 



Aesthetic Appeal 

This criterion covers issues of the physical quality of the product, 
the format, design and typology as well as the use of language, grammar 
and sentence structure. The aesthetic appeal issue should reflect the 
opinions and attitudes of students about using the product. 



Developmental Design 

This criterion relates to instructional methods, objectives and 
aesthetic appeal of the product. The design of the product sho'jld reflect 
care in the development of the content without calling attention to the 
design itself. 
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The social fairness and Intrinsic qualities criteria both require 
examination of the product Itself or careful documentation of the out- 
comes of usage, as in evaluation reports or learner verification and 
revision procedures* Judgments about these two clusters should be made 
on as much "hard" data as possible. 



Learner/User Effects 



For this criteria cluster, the reviewer must rely on data from ex- 
ternal sources Initially, since the product has not been used previously 
In the particular setting. Sources of such data include user reports, 
product catalogs and brochures from developers and publishers, or 
sponsors, etc. Ideally, a costly product is pilot-tested In a setting 
comparable to that of the intended users. 

Under the learning effects cluster, unintended positive or negative 
side effects should be evaluated as well as the intended cognitive, 
affective, psycho-social, or psycho-motor effects. Integral to the ex- 
amination of effects of the product is the analysis of the data describing 
these effects. 



Evidence of Product Effectiveness 



This criteria cluster addresses the issue of determining the effects 
realized and expected of products as well as assessing the credibility of 
the evidence provided to document the product's effectiveness. 

Coller (1976) defines effectiveness as follows: 

To determine the relative desirability of the product/ 
process in terms of the nature of effects, in respect 
to sum of effects and quality of effects; potency of 
effects, and meaningful ness of effects; cost/effectiveness, 
in respect to the ratio of the magnitude of effects to 
real costs, and interactive effects or, those effects that 
occur where the product/process comes into contact with a 
moderating variable. 

In other words, in assessing effects the reviewer must consider the nature 
of the effects, i.e., the cognitive, psychomotor, social and affective 
effects, realized by the product; the objectives sought by the user in 
selecting the product; the cost related to achieving the effects; the 
duration of the effects and the interactive effects. 

Schermer presents the following criteria for assessing effectiveness: 
1) strength of the effect, i.e., how well does the product meet its stated 
goals and objectives; 2) long term effects such as impact and retention of 
learned matter; 3) comprehensiveness or number of effects, 4) scope of the 
effect, i.e., how widely does the effect reach; 5) comparative effectiveness 
in terms of competitive products or alternative methods of achieving the 
user's objectives; and 6) efficiency of the use of learners' time. 
(Schermer, 1975, pp. 54-56.) 
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With these compatible criteria In mind, the product review In- 
strument of the study will examine the kinds of outcomes Intended by 
the product In terms of cognitive, affective, social and psychomotor 
effects; document the kinas of tools, procedures and strategies used 
by developers to establish the nature and level of their effects; and 
judge on the basis of the data provided whether the results achieved 
under test conditions are generallzable to other target groups, whether 
they can be attributed to the product as opposed to other Intervening 
variables, and whether the results have been objectively measured. 

« 

Careful Product Development 

Design/Development Procedures 

The criteria in this cluster relate to developer/publisher and 
sponsor responsibilities for using their expertise to develop quality 
products for use In the education market. Design/development procedures 
should be of such quality as to ensure their effectiveness to users. 
While methodology of research, development and evaluation In a linear 
progression may not be relevant to the consumer in terms of know-how, 
nor to developers In terms of time and resource constraints, the funda- 
mental Issues of adequate conceptualization and design of the product 
remain important. The Intrinsic qualities of the product depend upon 
it. Data from developers about the procedures they have employed should 
be used by reviewers as an effective gage of the adequacy of the product 
des i gn . 

Formative and Summative Evaluation 

The present study will document the evaluation activities of products 
included in the study. Formative evaluation will include formal and in- 
formal data-gathering which provides feedback to developers for the purpose 
of improving products. Summative evaluation is conducted by the developer, 
the sponsor, the publisher or the consumer after the product has been com- 
pleted and out of the hands of the developer. Data from such evaluations 
should provide information about the use of the product under normal field 
conditions, i.e., about the general izability of the product. 

Learner Verification and Revision 

In some places LVR has become a requirement for the adoption of 
products. The product review instrument will document the number of R&D 
products which have undergone that process. 

Expert Reviewers 

This criterion is included to determine whether the product has 
undergone a technical review using outside, qualified experts to judge 
the quality of the product during the formative stage of development. 
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Independent Evaluation 

Like summative evaluation, this criterion assesses what has been 
learned about the product from persons other than the developer. 

The issues under careful product development are included to 
document the kinds of procedures being used to determine the effectiveness 
of lab and center products. Few users have demonstrated an interest in 
the process of product development; however they are concerned with the 
outcomes. Developmental procedures are mainly of concern to developers 
and their sponsors. 
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APPENDIX B 

CRITERIA FOR SUBMISSION OF EDUCATIONAL MATERIALS 

TO 

EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) CLEARINGHOUSES 
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GUIDELINES AND RATING SCALE FOR EVALUATING DOCUMENTS 



Attached 1s a I0-po1nt rating scale used by the Clearinghouse director 
to evaluate and decide the disposition of documents acquired by the Clearing 
house. The rating scale also assists potential contributors In selecting 
documents for submission to the Clearinghouse. The rating scale Is based on 
three principal criteria: appropviatenesa ^ aooeaaibility^ and aignifioanoe* 



The first of th^s^^ appvopviateneaaf Is dealt with In the ERIC/CEM 
scope-note: 

We define educational management to Include all aspects 
of the administration, governance, and structure of public and 
private educational organizations at the elementary and secondary 
levels and the provision of facilities for their operation. We 
Intend to acquire relevant documents and journal articles that 
bear upon the theory of practice of administration and governance 
and that are generated from the fields of educational, public, 
and business administration and from the humanities and the 
social and behavioral sciences. 

Topics covered Include social, technological, political, and 
legal context of the organization; state, and federal programs and 
policies; methods and varieties or organizations, including various 
grade organizations, traditional and alternative schools, and re- 
scheduled school year plans; the tasks and processes of administra- 
tion, encompassing policy development and long-range planning, fi- 
nance, business management, law, public relations, staff personnel 
administration {e.g.» collective bargaining, evaluation, affirmative 
action), pupil personnel administration, conflict management, cur- 
riculum development, computer applications (e.g., data processing, 
simulation models), and quantitative methods (e.g., operations re- 
search, systems analysis) physical environment, including facility 
planning and design, construction, equipment and furnishings, and 
maintenance; and preservice and inservice preparation of adminis- 
trators. 

The Clearinghouse seeks for its collection not only typical research 
reports with their hypotheses, procedures, and findings, but also published 
and unpublished conference papers, speeches, essays, curriculum guides or 
studies, announcements of research in progress, interim project reports, 
books, bibliographies, and pther works of value for school administrators, 
professors, researchers, and the public. 



*ERIC Clearinghouse on Educational Management, University of Oregon, 
Eugene, OR. 
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The second criterion, aaaeosibiHtyt must be considered at almost 
every level of the rating scale. One of ERIC's primary objectives 1$ to 
make high quality material widely available as soon as possible. For a 
number of reasons, much good material remains unpublished, or Is published 
in local or state Journals with limited distributici, or is not published 
for several months, even years following completion. Submission of mate- 
rials to the Clearinghouse to be distributed through the ERIC Document 
Reproduction Service (EDRS) is one solution to this problem in the field 
of educational administration. The Clearinghouse also obtains for its 
local files all relevant materials that are widely disseminated through 
traditional channels. These documents are annotated and Indexed at the 
Clearinghouse and provide a resource pool for the preparing of its infor- 
mation analysis products. 

Signifiaanoe cannot, of course, be measured quantitatively without 
a much more sophisticated and detailed apparatus than is available. How- 
ever, it can be recognized; the director uses his experience and sub- 
stantive expertise to assign rankings to these documents. 
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Rating Scale for Evaluating Documents* 



1. Reroute to another clearinghouse. Not relevant to the domain of this 
Clearinghouse- 

2. Rejects Materials of nonprofessional standard, a;jpearing careless, 
naive, or the like; or an inhouse report so limited in scope or applica- 
tion that it would not be cf interest to anyone outside the author's 
school, districi:^ or state. 

3. Reject. Personal reminiscences, abstract generalities, or the like. 
Perhaps entertaining, but contains little information. 

4. Reject. Although not really a good piece of work, it does have t 
features (e.g., procedures, data) which could be of value to later 
researchers. 

5. Reject. Good work, but nothing new; a replication of previous well- 
known work; findings reinforce those of prior research. 

6. Reject. Good work, but the topic is of interest oniy to certain 
highly specialized members of the profession. 

7. Reject. Very good and of wide interest, but appearing in a high- 
visibility journal (i.e., a journal of high national circulation 
presumably availfeible to the majority of school administrators or 
professors). 

8. Accept for RIE. Good. Although a replication of previous work, 
findings differ from the earlier ones, or reinforce them for a dif- 
ferent population. Low visibility. 

9. Accept for RIE. A major new research project and/ or a major con- 
tribution to thought or knowledge of the field, but not yet pub- 
lished in a high-visibility journal. 

10. Accept for RIE. Because of its broad significance, this material 
should get the widest possible dissemination. 



^"Documents" include not only the research projects, but also conference 
papers, speeches, essays, curriculum guides and studies, re'^earch or 
progress analyses, books, bibliographies, and. other works of value to 
school administrators, professors, researcher is, and the public. 
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SUBMITTING DOCUMENTS FOR INPUT TO ERIC - SOME GUIDELINES FOR AUTHORS 



The ERIC system is designed to collect, index, and microfilm educational 
documents of use to teachers, administrators, researchers, and scholars. Many 
of the documents are not available from any source other than the ERIC system. 
The documents come in many different sizes, formats, and forms, from Ditto 
and Xerox copies to printed material. In dealing with the wide variety of 
documents which are submitted to the system, various reproduction problems are 
met which limit the readability and value of a number of these documents. 
In some cases, documents cannot be accepted into the system because of these 
reproducibility problems. 

Input to the system must be carefully monitored if a reasonable quality 
of document readability is to be maintained. A few basic fundamentals, if 
applied, can substantially enhance the overall quality and usefulness of the 
ERIC data base. Document producers, authors, and those submitting materials 
to ERIC, are advised to keep in mind the following principles: 

1. PAPER 

a. Weight of paper should conform as nearly as possible to that usually 
acceptable in business typewritten media. Medium weight bond or 
reproduction paper stock of 16 to 20 pounds is ideal. Use of onion- 
skin and other flim:>y or transparent types of paper should be avoided. 

b. Color of paper should preferably be white or a light tint, but 
should not include the darker shades or solid colors such as red, 
purple, orange, brown, blue, etc. Colored papers that reduce the 
contrast between the print and the background will not microfilm 
wel 1 . 

c. Size of paper is ideal when it is 8-1/2 x 11 inches. Larger sizes 
and foldouts will often create the need to make overlapping, multiple 
images of each page, with resultant viewing difficulties for the 
reader. 

2. TEXT ? 

Text of a document should be oriented, whenever possible, parallel to 
the short dimension of the paper. 

3. TYPE SIZE 

To insure acceptable reproduction in both microfiche and hard copy, 
minimum type size should oe 6-point type. In this size type, the height 
of the lower case letter "e" is approximately 1.6 millimeters. This is 
about the thickness of a f'ive-cent piece. The type font of a standard 
typewriter, pica^or elite, is well above the minimum acceptable size. 
Extremely narrow lines in drawings or type do not reproduce well and 
should be avoided. 
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TYPE DENSITY 

It is important that the density or "blackness" of the type be as great 
as possible- Original copies rather than copies should be submitted. 
In some cases, where the typewriter ribbon is old, even an original copy 
may be faint and may not reproduce well. In this special case, a Xerox 
copy may improve the type density. Purple "Dittos," colored inks, and 
blueprints, in general, will not reproduce satisfactorily. 

APPENDIX MATERIALS 

Because appendix materials are frequently reproduced from other sources, 
care should be taken to see that these materials are legible and meet 
the standards discussed above. If appendix materials are marginal, 
consideration should be given to omitting the questionable material. 
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APPENDIX C 

SPECIMEN JDRP SUBMISSION: 
San Diego State University Teacher Corps Project 

SYSTEM FOR INDIVIDUALLY GUIDING MASTERY ATTAINMENT 

(SIGMA) 

Approved by the JDRP on 8 May 1979 
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PROGRAM AREA: Teacher Corps - Competency*8ased Teacher Educatton 

PROJECT TITLE: SIGMA - System for Individually GuldInQ Mastery Attainment 

San Diego State University, San Oiego, California 

DEVELOPED BY: Staffs of Cycle VI and VIII - Teachers Corps Projects 

SOURCE AND LEVEL OF FUNDING: {The following are IHE budget totals onlyj 

U.S,O.E. Teacher Corps Cycle VI $2 ^9,482 

U,S,O.E Teacher Corps Cycle VIII $260,41 4 

YEARS OF INTERVENTION DEVELOPMENT: 

The development and field-testing perkxl was 1971 through 1975. 

BRIEF DESCRIPTION OF INTERVENTION: 

The objecth/e of this Intervention was to develop, field-test and evaluate a 31 credit hour, two semester, 
competency-based, preservice elementary teacher education program. The program was to be Individualized 
through modules (learning packages), personalized, team taught and organized by flow charted competencies 
rather than courses. 

Context and Students Served: 

San Diego State University is a large urban university where most students do not live on campus. The 
Department of Elementary Education is the largest of seven departments In the College of Education and employs 
approximately 40 full-time faculty. While there are a growing number of minority students In elementary 
education, the great majority are white, middle class females. Student teaching experiences are arranged so that 
each candidate is placed at least once In a racially or ethnically different environment 

Program Description: 

The content objectives of the SIGMA program are the same as those approved by the California State 
Commission for Teacher Preparation and Licensing for all other elementary credential sequences at San Diego 
State University. However, the organization and delivery of Instruction Is radically different. The content emphasis 
is not identified as course work, but flow-charted Into a skills development sequence intended for the most part to 
imparl generic teaching skills transferable to all settings. The selection and sequencing of these skills requires 
that the content be highly integrated, not only horizontally, but vertically between skill strands to establish 
prerequisites and interrelationships. 

Objectives generated from the Identification of skill areas are of four types: knowledge, performance, 
consequence, and affective. A concerted effort Is maintained to demand actual performance In realistic teaching 
situations. Since decisions on program development are made as a faculty team, duplications or omissions of 
content such as might be found In a typical series of courses are eliminated. 

Modules are used to Individualize the program so that students move through the skill development strands at 
their own pace. Each module Incorporates six characteristics: 

1) behavlorally stated objectives Including tennlnal behavtor. assessment conditions, and criterion level; 

2) behaviorally stated prerequisites for each module; 

3) a preassessment baaed on the objectives; 

4) a variety of learning alternatives teaching to each objective; 

5) a postassessment based on the objectives; and 

6) a provision for remediation if exit criterion Is not met on any objective.^ 

This Instructional delivery system utilizes 28 modules In the fall semester and 15 in the apring semester, and 
ehables the student to attain objectives In a variety of skill areas. The student through conferences and small 
group meetings. Is assisted by instructors to successfully complete the module requirements. Each student Is 
assigned to an on-site classroom in which the perfbrinance required by each module Is evaluated In a realistic 
educational setting. A concurrent seminar Is conducted throughout each semester to provide continuity and large 
group interaction, and to allow for further Instructor Input as needed. 

1 
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The foltowing figums depict the module sequence In each of the 



eemestera of SIGMA. 




MODULES 

1« Compttency-Bti^d 

Instruction 

ib-l InstructtontlObjtctivM 
R«inforc«mtntT«chniqu«i 

3 Motivation T«ctiniqu«t 

4 R«t«ntlon Strtttgitt 

5c-6 EnolishaaiLanouio* 
Syatam 

6 Linouao«Acqujifttontnd 
Usage 

Handwriting Proficiency 
Vocabulary Oavalopmani 
Childran'a Utaraturt 
Chifdran'a Intaraata 
Oral Reading 
I2a-b Decoding Skilla 
13 Comprehenalon 

Social Development 
Interaction Skilia 
Intellectual Skills 
Community Study Skilla 
Queattoning Skllla 
Concept Development 
Inquiry Proceaa 



7 
6 
9 

10 
11 



14 

15 
18 
17 
18 
19 
20 



FIGURE 1 



Flow Chart for Skill Development end Student TeecNng - Rret Semester (i 5 Weeks) 



WEEK*^'^** OiVELOPMENT 




STUDENT TEACHING 
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MODULES 

21 Reading Disgnoais 

22 Reading Approaches 

23 Reading Materials 

24 Educational 
Accountability 

25 Educational 
Measurement 

26 Long Range Planning 

27 Organizing for 
Instruction 

28 Learning Stations 

29 Writing a Module 

30 Creativity 

31 Creative Dramatics 

32 Creative Composition 

33 Creating with Art 

34 Creating with Music 

35 RolsPlsyln0 



FIGURES 



Flow Chert for Skill Development end Student Teeehing • Second Semeeter (1 5 Weeks) 
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The method and organizational system when compared to the noimai programs raqulros a number of 
modifications in the roies of sll participants The studenta are required to assume much morB responsibility for 
their own learning; the university Instructor is relieved of much of the task of Information delivery and becomes a 
manager counselor, and facilitator the cooperating classroom teacher becomes pivotal in providing 
opportunities for classroom experiences which help students bridge from theory to practice. The highly 
personalized approach to instnjctlon and the creation oi a noncompetitive learning environment results in the 
development of cooperative group behaviors. The positive group feeling creates an esprit de corps that keeps 
motivation high during the entire year^ 

Organizational structures chosen to deliver the model Involve a campus-based learning center and field-based 
laboratory. Campus facilities were provided in the form of an Individualized Study Center (ISC) to support the 
model with print^audio. and video media^ The SIGMA faculty team normally consists of five university faculty who 
work with a group of about 30-35 student teachers all year and are responsible for all skill development 
requirements and on-site supervision. 

Criterion-referenced assessments based on behavtorally stated objeetlvissare used to evaluate student teachers. 
The chteria stress attainmcint of mastery, and modules provide for remediation in order to reach that level. Should 
a student not attain mastery he/3he is provided with the opportunity to remediate the specific weakness which 
the instructor diagnoses. In most instances, all that the student needs in order to achieve mastery is more time to 
study or practice. If remediation is still needed at the end of a semester the student is given a grade of 
'Mncomplete.'* 

Evaluation of full-time student teaching Is highly correlated with the modular part of the program. During the first 
semester's field experience, each student is evaluated on the ability to integrate the previously developed 
competencies into his/her performance while working with a group of children. During the second semester full- 
time field experience, each student and cooperating teacher identifies the objectives that the student will try to 
accomplish with the children in the assigned classroom. The student develops and negotiates a contract with the 
cooperating teacher and university supervisor which specifies certain desired conseQuences from working with 
the Children. The final evaluation is based on the student's demonstrated effectiveness in fulfilling this 
accountability contract 

in order to make SIGMA a regenerative program, responsive to students and changing perspectives, both 
summative and formative data are collected periodically. 

EVIDENCE OF EFFECTIVENESS: 

In the 1 974-1 975 school year Donald F. Enos collected data on the SIGMA program and a control group, and in 
1 975- 1 976 he did a follow-up study of employed graduates of both programs, analyzed data, and documented the 
results.^ He was assisted with data collection from time to time by graduate students from San Diego State. 
iHowever none of those involved in conducting the research were at any time Involved with the development or 
implementation of the SIGMA program. Additionally, no member of the SIGMA faculty team or any other faculty or 
students were involved with the specification of hypotheses, data collection, analysis, or reporting results and 
conclusions. 

The SIGMA program (N-33) was conducted by three faculty plus two student teaching supervisora while the 
control group program (N«-40) was run by twelve faculty and six supervisors. The treatment for the control group 
was the regular instructional program in elementary education consisting of a foundations course and several 
methods courses in a variety of aubject areas. The number of credit hours In each program was equal. Control 
group students were taught mostly through lecture and discussion, and evaluations were of a norm-referenced 
variety. 

It was not possible, because of administrative constraints, to randomly assign student teachers to the two 
programs, but it Is believed that the groups were comparable since students were not selected especially for 
either program. All were enrolled on the basis of the regular registration procedure used by the Department of 
Elementary Education. Additionally, no significant differences were found when comparisons of pre-enrollment 
interview ratings. GPAs, or numbera of men and women were made for the two programs. It should also be noted 
that students were observed to be more reluctant to enter the SIGMA program since it was an unknown and, in 
fact, the SIGMA block was the last to fill at registration out of s total of eight blocks available. As a result, it is 
believed that the two groups of students were as equivalent as they would have been had it been possible to make 
random assignments. 
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Ctalmt of EffMttvMmt: 

^nc. .««=„»„«i. „„ b. p™.e„,„ ,„ ,o„ ,^ Whan c,mp.™d .„h . conw, ,„„p, s,g*« 



1. 



2. 



•moloxd vaduatas. ' ° ewtlllcallon pnsgrama and In a folloo-up study of 

Biidy M .mploy«) ,rKl° .l« I™""™"!. «■ bom durin, ina cartlllcatlon p,og™„a and In a follow-up 
fir^lnrg^t^rZi'^^^^^^^^^ ... SIGMA p,o„a™ *n,on.,™,«, . 

-ucafion a. san pi.^ s,a.a '^'''^^^^K.;!:^:s^^^^zr^j?:„v£r^ 

apptovad by tna California sita Smm^fJI:?,, ? I 2 D»Partmanl of Elamantaiy Eduoallon ProgVao, 
raliabillty was found to be 0.79 for tna a'anT oblactlvat Kuder-Blehaniaon formula 21 

.^|f^n.ly,raat.at..,n™nto,tna<S^!:S:Sl<^,r^^^^^^^^^ 

TABLE I 
_ Results for 

Studont Knowlsdgs Asssssmant of Tsaehsr EdueaUon 



SEMESTER 


SIGMA ( 
MEAN 


N-33) 
SO 


RTE (N 
MEAN 


-40) 
SO 


df 


t 


Fall 

Spring 

Year 


11.91 
10.33 
22.24 


2.01 
2.99 
3.68 


5.50 
6.95 
12.85 


2.72 
4.41 
5.37 


71 
71 
71 


11.25* 
3.52* 
11. 89* 


• SignificMnt dithmnce In /avor 0/ SiQMA gt th0 .001 /avi 
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The ReciprocMl Category System for verbal Interaction analyais waa developed by Richard Ober ® Thia ayatem 
waa uaed to code verbal Interaction into the following five categoriea: a) Warm/Cool Ratio, b) Acceptance/ 
Correction Ratio, c) Ellclt/lnltlate Ratio, d) Chlldren'a TaJk/Student Teacher Talk Ratio, and e) Acceptance/ 
Rejection Ratio. Inter-rater reliability waa found to be .91 before data collection began. 

Significant differences favoring the SIGMA Program were found In 1 7 out of 20 Chi Square teats of data on verbal 
Interaction aa meaaured by the Reciprocal Category System (see Tabie II). The atudent teachera of the SIGMA 
program were able to demonatrate algnlflcantly better verbal Interaction with children than were student 
teachers In the RTE program. The differencea noted were aa follows: 

r The ability to provide for and personally ntilize more poaltlve reinforcement and the elimination of tension 
within the classroom. 

2. The ability to provide a more accepting classroom atmoaphere and to facilitate the development of more 
poaitive feelings within the chlldr^^n. 

3. The ability to provide increased opportunities for children to present unsolicited facts. Information, and 
opinions during the instructional process. 

4. The ability to facilitate the chlldren'a Interactions within the classroom. 

TABLE II 
Results for Verbal Interaction 
Meaaured by the Reciprocal Category System 



Category 


Measuremefit 


Ratio Achieved 






Period 


SIQiyiA 


RTE 




df 


Warm/Cool 


Year'74--75 


5.43 


0.55 


361.66' 






Fail '74 


4.12 


2.62 


4.07" 






Spring '75 


7.03 


0.25 


432.92- 






Follow-up *75 


45.60 


0.25 


123.16' 




Accept/Correct 


Year 74-75 


9.73 


2.11 


277.18' 






Fall 74 


6.95 


1.91 


156.35' 






Spring 75 


10.61 


2.52 


100.39' 






Follow-up 75 


13.33 


1.03 


73.56' 




Elicit/Initiate 


Year 74-75 


1.01 


.79 


74.46' 






Fall 74 


1.35 


1.21 


2.26 






Spring 75 


.65 


.51 


61.67' 






Follow-up 75 


.66 


.62 


1.93 




Student Teacher Talks/ 


Year 74- 75 


1.22 


1.95 


533.17' 




Children Talk 


Fall 74 


1.44 


1.91 


98.46' 






Spring 75 


1.05 


1.96 


506.67' 






Follow-up 75 


1.19 


1.31 


1.57 




Acceptance/Rejection 


Year 74- 75 


9.60 


1.69 


565.39' 






Fall 74 


9.01 


2.94 


132.64' 






Spring 75 


10.72 


1.06 


566.33' 






Follow-up 75 


44.23 


1.02 


222.21' 





' Significant difference In fevor of S/G/UA at the .001 leyel. 
^ Significant difference In feyor of S/G/UA et the .05 level. 



Aa noted above. afollo^^Nup atudy of graduatea of both programa waa done during fall 1975. Only six of the 40 
RTE program graduates and 12 of the 33 SIGMA graduates had found employment by that time Results from 
euch a small sample cannot be conclualve; however, they can provide some Indication as to whether or not the 
findings of the year-long atudy are maintained. It la believed that the follow-up atudy reaulta help to support the 
validity of the concluslona of the basic atudy. 
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atlm of EffMlivM«88 3: When comptrod with t control group, tho SIGMA program domonatrBtoa 
significantly more use of Individualized Instruction by student teachers when working with children aa 
measured by the Descriptive ObservstionsI Record for Indlvldusllzed Instruction snd the Individualized 
Inatructlon Inventory, both during the certification programs and In a follow*up study of employed grsduates. 

It had been the intention of the SIGMA developers to model a type of Individualized Instruction which would be 
emulsted by student teachers In their work with children. The competency-based Instructional approach la a 
particular type of Mastery Learning, and revlewa of research on Maatery Learning by Block^ and others have 
shown dramatic achievement gains by pupils. In summsry. Therese M. Herman* concludes that research with 
individualized instruction of all kinds clearly demonstrstes thst many students who use It lesm at least aa well, 
sometimes better, and often, though not aJways, more quickly than they leem with other kinds of Inatructlon. 

The Instruments chosen to sssess student tecchem* degree of IndlvMuallzing were the Descriptive 
Observetional Record for Irtdlvlduellzatlon of Irtetructlci (DORII) by Bon Harris snd Ken Mclntyr^ snd the 
Indlviduelizatior^ of Irtstructlon Ir^ventory (m) by Betty Coody snd Ben Hsirls-^These Instruments grouped data 
into the following factors; a) Intra-Claaa Grouping, b) Variety of Materials, c) Pupil Autonomy, d) Oih'erentlated 
Assignments, and e) Tutoring. Inter-observer reliability was found to be £9 before d^ts collection becan. 

Significant differences at the .001 level were found for Individualization of instruction measured by DOR II &nd 111 
In all but the categories of Intra-Class Grouping and Variety of Materiala for tha fall semester, 1974 (see Table lii>. 
The SIGMA program provUed an increased number of training techniques for the etudent teachers In 
individualizing instruction within the classroom. The SIGMA progrsm student teachers provided more flexibility 
snd direct taskHDriented situatlona for children; a wider variety of available materials throughout the classroom: 
and greatiK pupil participation In plannlna eeif-dlrectton, leadership In groups, and IndlvldusI activities. The 
SIGMA program student teachers also provided greater challenges In children's assignments, as well as 
relationship of the assignments to diagnosed learning needs, and tutoring of Individuals with learning problems. 



TABLE III 

Results of Indhrlduellxatlon of Instruction 
Messured by OOR II snd III 





Meesurenient 


StQMA 


RTe 






Category 


Period 


Mean 


8D 


Mean 


SD 


df 


t 


Intra-Clsss GroupinQ 


Year '74^75 


19.12 


1.74 


17.38 


1^9 


71 


4.07* 




FsU 74 


18.57 


2.37 


1&37 


2.18 


71 


1.75 




Spring 75 


1957 


2.82 


18.35 


3.01 


71 


4.78- 




Follow-up 75 


2^30 


1.43 


17.00 


2.12 


12 


7.46« 


Vsritty of l^ateriais 


Year 74^75 


17.91 


1.51 


14.95 


1.33 


71 


8.93* 




Fall 74 


15.58 


2.09 


15.45 


2.05 


71 


0.28 




Spring 75 


19.24 


2.92 


13.45 


1.78 


71 


12.08* 




Follow-up 75 


21.70 


1-87 


15.00 


1J7 


12 


7.22* 


Pupil Autcnomy 


YMr 74-75 


17.33 


1.59 


15.15 


1.33 


71 


8.39* 




Fall 74 


17.24 


2.11 


15.35 


1.83 


71 


4.33* 




Spring */5 


17.42 


2.59 


15.03 


1.88 


71 


4.48* 




Follow-up 75 


20.30 


0.82 


15.80 


1.48 


12 


10.00* 


Olffer»r,tiated 


Year 7'»-.75 


18.48 


t.47 


If 08 


1.43 


71 


9.89* 


Assignments 


Fall 7^1 


17^9 


1.87 


15.75 


1.72 


71 


3.91* 




Spring 75 


19.45 


2.82 


14.40 


1J9 


71 


9.34* 




Follow-up 75 


21.70 


1.41 


15.25 


3.03 


12 


8.19* 


Tutoring 


Yoar 74-75 


17^4 


1^9 


14.30 


1.73 


7t 


8.6*3* 




Fan 74 


17.97 


3.01 


15.08 


^02 


71 


4.90* 




Spring 75 


17^1 


2.72 


ia55 


I*.43 


71 


7.23* 




Follow-up 75 


20.50 


2.00 


15.00 


1.36 


12 


7.00* 


ToUl Indlvldusilzatlon 


Year 74-75 


90.73 


8.91 


75.75 


5.&8 


71 


9.28* 


of Instruction 


Fail 74 


87.75 


9.84 


81.13 


7.24 


71 


3.31* 




Spring 7,5 


93.89 


10.98 


72.77 


9.49 


71 


8.74- 




Follow-up 75 


105.90 


4.91 


80.00 


9.11 


12 


9.71* 



* Slgnfflctnt dimnnc9 In ftvor ot SlQUA »t .001 l0v»L 



6 

78 
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While th« sample size for the follow-up study Is small. It does provido results consistent with those found during 
the earlier periods of student teaching. 

Claim of Effactlveneea 4: When compared with a control group, the SIGMA program demonstrates 
significantly higher ratings of student teacher performance by the children taught as measured by ihe 
Student Evaluation of Teacher Instrument II, both during the certification programs and In a follow-up study of 
employed graduates. 

Students' attitudes toward their teacher are often used In the profession as one Index of teaching 
effectiveness. However, according to Getzels and Jackson, research provides mixed support for a direct lli^k 
between pupil attitudes and outcome measures such as achievement^^ Even so. It Is believed to be both 
sensible and desirable for a teacher to strive toward being liked by pupils as long as Instruction quality Is not 
sacrificed. 

The instrument selected to sample chlldrens' attitudes was Student Evaluation of Teacher Instrument II (SETII) 
by Ruth Haak, Douglas Klelber and Robert Peck. Children's attitudes toward their student teacher /ere 
sampled for each of the following factors: a) Interactional Competence, b) Rapport, c) Stimulating, Interactive 
Style, d) Unreasonable Negativity, and e) Fosterance of Self-Esteem. Published reliabilities indicate that r - .75 
for kindergarten, t - .82 for grades 1 --3, and r .78 for grades 4—6. 

Each student teacher's overall perfonnance was rated by the children they had wortced with using the SETII. A 
grand total of 2,01 9 children rated the SIGMA student teachers and 2,248 children rated the student teachers in 
the control group. For the spring semester and the composite analysis for the 1974-1975 academic year, 
significant differences were consistently found favoring the SIGMA program except for tho kindergarten classes 
(see Table IV). 

Only one significant difference In favor of the RTEpro^pam was found ("Unreasonable Negativity," fall semester, 
grades 1 -3). Subsequent Investigation and analysis reversed that finding In the spring semester. 1975. and again 
in the follow-up study during the fall semester 1975. These analyses suggested that the SIGMA program provided I 
better opportunities for the student teachers to be understood by the children in the classroom. 

When examining Table iV. the null hypothesis must be accepted for the fall semester because there Is a 
consistent pattern of no significant difference In children's ratings of student teacher performance. However, 
during the measurement periods of spring 1975 and follow-up 1975. significant differences In ratings of the 
children were observed favoring the SIGMA program student teachers In the areas of stimulating. Interactive 
style, rapport Interacttonal competence, unreasonable negativity, and the fosterance of self-esteem. 

Controls for Internal Validity: 

Threats to the Internal vaildify of the study were controlled as follows: 

1. History Effect: SIGMA AND RTE programs were conducted In the same time period tr\6 thus were subjecf: 
to the same. external forcea 

2. Maturation Effect: The use of a control group helps to limit any possible maturation effect, but there is a 
possibility of a Hawthorne Effect However, the SIGMA program consists of a year of very hard, full-time 
work; and this combined with the results of the follow-up study tend to make it appear unlikely that a 
Hawthorne Effect was operating. 

3. Testing Effect: If there were any testing effect it would have worked equally en each prograjn group. 

4. Instrumentation: Reliability and validity for instruments and observers were substantiated and/or 
maintained at high levels. 

5. Statistical Regression: No student teachers were selected for their extreme scores. 

6. Mortality: No students were lost from either SIGMA or the RTE program groups. 

7. Selection Effect: Since random assignment of students to each program was not possible, there Is a 
chance of a selection effect, but as was noted eariler students did not differ significantly in interview 
ratings. GPA, or numbers of men and women. Students chose the program they wished, and SIGMA was { 
the last program to fill at registration. Due to these circumstances It Is believed the groups were equivalent 

8. Selection-Maturation Interaction: It is believed this was controlled as noted In numbers 2 and 7 above. 
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TABLE IV 

RmuIIs of Student E«alMtlon of To«eh«r bistrumont II 



Factor 


MaAtt-UTMlMt 




tlQMA 


ute 






Period 


Qrada 






MMn 




df 


t 




Year *74— *75 


K 


9.61 


0.37 


10.17 


0.64 


24 


2.24*** 


IntArs /"lit/A C^uIa 
inivraciivo Qiyw 


Fall '74 


K 


10.17 


0.39 


9.97 


0.56 


11 


0.89 




Spring *75 


K 


O 79 




1 U.0 1 


0.63 


lU 


3.91 




Folio w*up 75 


K 




w. 1 O 


Ma »»m4 
no iM4 


inWfMmlltUi 






Year 74-75 


1-3 


9.64 


0.50 


10.52 


0.76 


58 


7.38* 




Fall 74 


1-3 


9.67 


0.63 


10.23 


0.76 


26 


1.45 




opring / s 


1-3 


9.46 


0.26 


10.75 


0.75 


14.37 


5.96* 




Follow-up 75 


1-3 


9.36 


0.14 


10.56 


0.41 


6 


8.13* 1 


Development 


Year 74-75 


4-6 


5.47 


0.37 


5.92 


0.64 


70 


5.12* 1 


of Rapport 


Fall 74 


4-6 


5.59 


0.41 


5.76 


0.63 


30.89 


1.03 




Spring 75 


4-6 




O 1 7 

U.I f 




A A^ 


97 4 A 

2r .1 0 


9.94 1 




Folk) WHIP 75 


4-8 




O 09 


A 77 


o aa 


2 


1 25.00* 


tntenctionai 


Year 74—75 


4-6 


4.43 


0.25 


4.69 


0.45 


70 


j 

4.29* 


Com pet en c 6 


Fall 74 


4-6 


4.53 


0.24 


4.57 


0.50 


25.72 


0.35 




Spring 75 


4-6 


4.27 


0.16 


4.78 


0.36 


**** 1 7 


K Ay 1 
0.4 1 1 




Follow-up 75 


4-6 


4.09 


0.01 


4.94 


2.22 


2 


70.83* 1 


UnreasonahlA 


T Oar / #— f O 




9.32 


0.69 


8.86 


1.11 


24 


1 

1.65 1 




Pall *7A 
roll / # 




7.98 


0.46 


6.78 


1.35 


11 


1.58 1 




Spring 75 


K 


9.66 


0.10 


6.96 




3.07 


2 17 1 




Follow-up 75 


K 


9.80 


0.03 


Mo fMC 










Year 74—75 


1-3 


9.34 


0.46 


6.86 


0.96 


58 


1 

3.51* 




Fall 74 


1-3 


8.91 


0.37 


9.47 


0.41 


26 


3.83 f 




Spring 75 


1-3 


9.68 


0.1 7 


8.18 


1.01 


1 9 AQ 


^ 9fl* 1 




Follow-up 75 


1-3 


9.91 


0.08 


670 


0.42 


6 


8.07* I 




Year 74— 75 


4-6 


9.40 


0.51 


6.94 


0.75 


70 


J 

4.21* 




Fall 74 


4-6 


9.26 


C.45 


9 25 


0.74 


29.29 


0.17 




Spring 75 


4-6 


9.58 


0.58 


8.61 


0.74 


34 


3.2^** 




"qMow-up 75 


-4-6 


9.66 


0.34 


7.97 


2.82 


2 


19.88** J 




Toar t lo 


1^ 
P\ 


6.fr 1 


0.33 


6.64 


0.72 


24 


1 

1.10 


wVll C9IOVM1 


Pall *yA 

rail f # 


1^ 
i> 


7 1 1 
r . 1 9 1 




A TO 


U.93 


11 


1.87 




Spring 75 


K 


6.5t 1 


0.26 


7.05 


0.57 


3.63 


1.67 




Follcw-up 75 


K 


6.39 


0.04 






1 






Yaar 74-75 


1-3 


6.62 


0.2^ 


7.09 


0.22 


58 


5.33* 




Fall 74 


1-3 


6.98 


0.54 


666 


0.52 


26 


0.50 




Spring 75 


1-3 


6.34 


0.19 


7.24 


0.36 


16.26 


7.65* 




Follow-up 75 


1-3 


6.29 


0.10 


7.40 


0.08 


6 


20.94* 




Yaar 74-75 


4-6 


6.97 


0.44 


7.52 


0.60 


70 


4.46* 1 




Fall 74 


4-6 


7.33 


0.61 


7.23 


O.CJ 


37 


0.47 




Spring 75 


4-6 


6.41 


0.28 


7.65 


0.75 : 


30.75 


7.11* 


1 


Follow-up 75 


4-6 


6.4. 


0.34 


990 


2.98 


2 


27.87** 



« SIgnlflcMnt diff9r9nc0 In fvor of SIQMA st .OOf /ei^e/. 
** Significant dWerence In f9vor of SIQMA er .07 ImL 
Significant dlffennc^ In f^vor of SIQMA e/ .05 fvL 
f Significant dWere/ice In favor ofRTE er .ODf /ei^e/. 
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Costs for Adoption and Mslntonanco: 

In 1 976 oosts were as follows:^ 



^ensoNNCL 



INSTALLATION 



SUBSEQUENT 
YEARS 



1. Training Consuttants 

2. Faculty Team 

3. tSC Supervisor 



$1*225 
9.618 
4.625 



0 

Same aa Regular Program 
$9,250 



INDIVIDUAUZEn STUDY CENTER 



1. Facility Development 

a. Furniture 

b. AV Equipment 

c. Remodeling 



3.500 
4.250 
4.000 



0 
250 
0 



2. Software for Modules 



$3,062 



150 



CONSUMABLES - 35 sets of modules for students 



412 



Paid by Student Fee 



TOTALS 



$30,692 



$9,650 



CONCLUSIONS: 

The Enos study has presented very strong support for the effectiveness of the SIGMA program and establishes It 
clearly as a viable alternative to regular programs of elementary teacher preparation. The study has demonstrated 
that student teachers completing the SIGMA program obtained the following benefits, compared with student 
teachers completing the regular program: 

1. Significantly greater knowledge about teaching and lek ning, 

2. Significantly better verbal Interaction with childron, 
a Significantly greater use of Individualized Instruction, 

4. Significantly higher ratings of their performance from children they taught 

7h e outcomes of this study seem to be consistent with those being achieved with Mastery Learning in a variety of 
other subject matter areas.^ Additionally, SIGMA appears to be generallzable and dissemlnable. It Is believed that 
the dissemination to adopters should Include careful counseling and training, systematic assembly of modular 
components, preparation of an Individualized Study Center, and follow-up visits from consultants. All of these 
requirements appear ;o be realistically feasible for dissemination and Implementation at other sites in IHEs, LEAs, 
or even some county or state offices should such desire to credential preservlce elementary teachers. Since most 
of the competencies are generic In nature, no conflict is anticipated with state laws; however, states may have 
additional requirements beyond SIGMA which could be handled in a traditional manner. 

It Is believed that SIGMA is a dissemlnable program which has been successful In raising the quality of new 
teachers entering thu profession, while at the same time providing a model for teachers to emulate In working with 
children. 
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